From 66e362d96df8c5d8d9170ae4c871047c4c23acfc Mon Sep 17 00:00:00 2001 Message-Id: <66e362d96df8c5d8d9170ae4c871047c4c23acfc.1334848702.git.minovotn@redhat.com> From: Jason Baron Date: Wed, 18 Apr 2012 18:23:03 +0200 Subject: [PATCH 1/2] device assignment: Disable MSI/MSI-X in assigned device reset path RH-Author: Jason Baron Message-id: <201204181823.q3IIN3v9003219@int-mx12.intmail.prod.int.phx2.redhat.com> Patchwork-id: 39321 O-Subject: [RHEL 6.3 qemu-kvm PATCH] device assignment: Disable MSI/MSI-X in assigned device reset path Bugzilla: 798967 RH-Acked-by: Michael S. Tsirkin RH-Acked-by: Alex Williamson RH-Acked-by: Don Dutile Hi, Backport that Alex and I worked on with small changes for rhel6. Marcelo indicated that this was pulled but I didn't see the patch in qemu-kvm yet, so I don't have a commit id. Here's a pointer to the upstream posting/discussion: http://marc.info/?l=kvm&m=133359744225083&w=2 Also, although I've done a lot of testing with very similar code, I didn't test this exact patch yet on the affected system, since its currently unavailable. I'm trying to reserve it again, and will follow-up with further testing results. Posted today to get in before the deadline. Should resolve bz #798967. Brew build: https://brewweb.devel.redhat.com/taskinfo?taskID=4312161 Upstream: 9059b2561d4f33c8d2079aeae84e723901731b5a Thanks, -Jason We've hit a kernel host panic, when issuing a 'system_reset' with an 82576 nic assigned and a Windows guest. Host system is a PowerEdge R815. [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 32993 [Hardware Error]: APEI generic hardware error status [Hardware Error]: severity: 1, fatal [Hardware Error]: section: 0, severity: 1, fatal [Hardware Error]: flags: 0x01 [Hardware Error]: primary [Hardware Error]: section_type: PCIe error [Hardware Error]: port_type: 0, PCIe end point [Hardware Error]: version: 1.0 [Hardware Error]: command: 0x0000, status: 0x0010 [Hardware Error]: device_id: 0000:08:00.0 [Hardware Error]: slot: 1 [Hardware Error]: secondary_bus: 0x00 [Hardware Error]: vendor_id: 0x8086, device_id: 0x10c9 [Hardware Error]: class_code: 000002 [Hardware Error]: aer_status: 0x00100000, aer_mask: 0x00018000 [Hardware Error]: Unsupported Request [Hardware Error]: aer_layer=Transaction Layer, aer_agent=Requester ID [Hardware Error]: aer_uncor_severity: 0x00067011 [Hardware Error]: aer_tlp_header: 40001001 0020000f edbf800c 01000000 [Hardware Error]: section: 1, severity: 1, fatal [Hardware Error]: flags: 0x01 [Hardware Error]: primary [Hardware Error]: section_type: PCIe error [Hardware Error]: port_type: 0, PCIe end point [Hardware Error]: version: 1.0 [Hardware Error]: command: 0x0000, status: 0x0010 [Hardware Error]: device_id: 0000:08:00.0 [Hardware Error]: slot: 1 [Hardware Error]: secondary_bus: 0x00 [Hardware Error]: vendor_id: 0x8086, device_id: 0x10c9 [Hardware Error]: class_code: 000002 [Hardware Error]: aer_status: 0x00100000, aer_mask: 0x00018000 [Hardware Error]: Unsupported Request [Hardware Error]: aer_layer=Transaction Layer, aer_agent=Requester ID [Hardware Error]: aer_uncor_severity: 0x00067011 [Hardware Error]: aer_tlp_header: 40001001 0020000f edbf800c 01000000 Kernel panic - not syncing: Fatal hardware error! Pid: 0, comm: swapper Not tainted 2.6.32-242.el6.x86_64 #1 Call Trace: [] ? panic+0xa0/0x168 [] ? ghes_notify_nmi+0x17c/0x180 [] ? notifier_call_chain+0x55/0x80 [] ? atomic_notifier_call_chain+0x1a/0x20 [] ? notify_die+0x2e/0x30 [] ? do_nmi+0x1a1/0x2b0 [] ? nmi+0x20/0x30 [] ? native_safe_halt+0xb/0x10 <> [] ? default_idle+0x4d/0xb0 [] ? cpu_idle+0xb6/0x110 [] ? rest_init+0x7a/0x80 [] ? start_kernel+0x424/0x430 [] ? x86_64_start_reservations+0x125/0x129 [] ? x86_64_start_kernel+0xfa/0x109 The root cause of the problem is that the 'reset_assigned_device()' code first writes a 0 to the command register. Then, when qemu subsequently does a kvm_deassign_irq() (called by assign_irq(), in the system_reset path), the kernel ends up calling '__msix_mask_irq()', which performs a write to the memory mapped msi vector space. Since, we've explicitly told the device to disallow mmio access (via the 0 write to the command register), we end up with the above 'Unsupported Request'. The fix here is to first disable MSI-X, before doing the reset. We also disable MSI, leaving the device in INTx mode. In this way, the device is a known state after reset, and we avoid touching msi memory mapped space on any subsequent 'kvm_deassign_irq()'. Thanks to Michael S. Tsirkin for help in understanding what was going on here and Jason Baron, the original debugger of this problem. Signed-off-by: Alex Williamson Signed-off-by: Jason Baron --- hw/device-assignment.c | 26 +++++++++++++++++++++++++- 1 files changed, 25 insertions(+), 1 deletions(-) Signed-off-by: Michal Novotny --- hw/device-assignment.c | 26 +++++++++++++++++++++++++- 1 files changed, 25 insertions(+), 1 deletions(-) diff --git a/hw/device-assignment.c b/hw/device-assignment.c index 3cdce01..c1150f4 100644 --- a/hw/device-assignment.c +++ b/hw/device-assignment.c @@ -1719,7 +1719,31 @@ static void reset_assigned_device(DeviceState *dev) AssignedDevice *adev = DO_UPCAST(AssignedDevice, dev, pci_dev); char reset_file[64]; const char reset[] = "1"; - int fd, ret; + int fd, ret, pos; + + /* + * If a guest is reset without being shutdown, MSI/MSI-X can still + * be running. We want to return the device to a known state on + * reset, so disable those here. We especially do not want MSI-X + * enabled since it lives in MMIO space, which is about to get + * disabled. + */ + if (adev->irq_requested_type & KVM_DEV_IRQ_GUEST_MSIX) { + if ((pos = pci_find_cap_offset(pci_dev, PCI_CAP_ID_MSIX, 0))) { + uint16_t ctrl = pci_get_word(pci_dev->config + pos + + PCI_MSIX_FLAGS); + pci_set_word(pci_dev->config + pos + PCI_MSIX_FLAGS, + ctrl & ~PCI_MSIX_FLAGS_ENABLE); + assigned_dev_update_msix(pci_dev, pos + PCI_MSIX_FLAGS); + } + } else if (adev->irq_requested_type & KVM_DEV_IRQ_GUEST_MSI) { + if ((pos = pci_find_cap_offset(pci_dev, PCI_CAP_ID_MSI, 0))) { + uint8_t ctrl = pci_get_byte(pci_dev->config + pos + PCI_MSI_FLAGS); + pci_set_byte(pci_dev->config + pos + PCI_MSI_FLAGS, + ctrl & ~PCI_MSI_FLAGS_ENABLE); + assigned_dev_update_msi(pci_dev, pos + PCI_MSI_FLAGS); + } + } snprintf(reset_file, sizeof(reset_file), "/sys/bus/pci/devices/0000:%02x:%02x.%01x/reset", -- 1.7.7.6