FastNetMon

понедельник, 6 июня 2016 г.

Debian, KVM and PCI-E passthrough for Intel 82599 / Intel X-540 NIC's

First of all, please enable hardware virtualization and VT-d (I/O virtualization) in BIOS. It's mandatory!

Install KVM tools and distro install tools on the Debian:
apt-get install -y qemu-kvm  libvirt-daemon libvirt-clients libvirt-daemon-system virtinst
Enable libvirt and libvirt.guests services:
systemctl restart libvirtd.service
systemctl restart libvirt-guests.service 
Run default network:
virsh net-autostart default
virsh net-start default
Install Debian Jessie:
virt-install --location http://ftp.us.debian.org/debian/dists/stable/main/installer-amd64/ --ram 16000 --file  /var/lib/libvirt/images/fastnetmonvm.fastvps.ru.disk  --file-size 20 --name fastnetmonvm.fastvps.ru -x console=ttyS0 
Switch on autostart for VM:
virsh autostart fastnetmonvm.fastvps.ru 
Well, we should pass whole NIC to VM.

List all available NIC's on the server:
lspci |grep Ether
03:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
We will pass two NICs (it's two port NIC card):
03:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Get PCI device names in virsh format (03_00_0 created from 03:00.0 with changing ':' by '_'):
virsh nodedev-list | grep pci |grep 03_00_0
pci_0000_03_00_0
pci_0000_03_00_1
Get XML descriptions for both PCI devices:
virsh nodedev-dumpxml pci_0000_03_00_0
virsh nodedev-dumpxml pci_0000_03_00_1 
Switch off drivers for this NIC's on server:
ifconfig ethX down
rmmod ixgbe
Detach both NIC from the Linux:
virsh nodedev-dettach pci_0000_03_00_0
virsh nodedev-dettach pci_0000_03_00_1
If you got: "error: Operation not supported: neither VFIO nor KVM device assignment is currently supported on this system".

Well, we should enable IOMMU for fixing this issue:
vim /etc/default/grub
Change following line to:
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"
Apply grub settings:
update-grub
And reboot server:
shutdown -r now
Then repeat last steps.

List devices from virsh:
virsh nodedev-dumpxml pci_0000_03_00_0 |grep address
   <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
   <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> 
Then please build something like this using data above:
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
      <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
  </source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
      <address domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/>
  </source>
</hostdev>
Then open VM edit and insert this block to "devices" block:
virsh edit fastnetmonvm.fastvps.ru
Then you could try to run VM:
virsh start fastnetmonvm.fastvps.ru
You could get error similar to this:
error: Failed to start domain fastnetmonvm.fastvps.ru
error: internal error: early end of file from monitor: possible problem:
2015-05-13T13:12:39.353614Z qemu-system-x86_64: -device vfio-pci,host=0a:00.0,id=hostdev0,bus=pci.0,addr=0x6: vfio: failed to set iommu for container: Operation not permitted
2015-05-13T13:12:39.353761Z qemu-system-x86_64: -device vfio-pci,host=0a:00.0,id=hostdev0,bus=pci.0,addr=0x6: vfio: failed to setup container for group 23
2015-05-13T13:12:39.353780Z qemu-system-x86_64: -device vfio-pci,host=0a:00.0,id=hostdev0,bus=pci.0,addr=0x6: vfio: failed to get group 23
2015-05-13T13:12:39.353804Z qemu-system-x86_64: -device vfio-pci,host=0a:00.0,id=hostdev0,bus=pci.0,addr=0x6: Device initialization failed.
2015-05-13T13:12:39.353827Z qemu-system-x86_64: -device vfio-pci,host=0a:00.0,id=hostdev0,bus=pci.0,addr=0x6: Device 'vfio-pci' could not be initialized
If you have something like " vfio-pci 0000:0a:00.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor." in dmesg that's not a good news for you. You could not use IOMMU for your server and need to buy another.

For fix issue "vfio_iommu_type1_attach_group: No interrupt remapping support" please use:
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf
And reboot again.

Sources:   https://www.paloaltonetworks.com/content/dam/paloaltonetworks-com/en_US/assets/pdf/framemaker/61/pan-os/NewFeaturesGuide/section_7.pdf and http://h20564.www2.hp.com/hpsc/doc/public/display?docId=emr_na-c03645796

 

1 комментарий :