FastNetMon

Tuesday 16 June 2015

How to run netmap on ixgbe Virtual Function on Debian 8 Jessie

UPDATE: folks, please be AWARE! In this mode netmap working only in single-copy mode and will not work enough fast (1-2 mpps is a limit). For full support we need another patch for ixgbevf driver.

If you want zero copy support, please poke netmap guys here: https://github.com/luigirizzo/netmap/issues/63

First of all, install Netmap and ixgbe module: http://www.stableit.ru/2014/10/netmap-debian-7-wheezy-intel-82599.html

Please add this code to /etc/rc.local (please remove -e flag from /bin/sh in first line of /etc/rc.local):
insmod /usr/src/netmap/LINUX/netmap.ko
modprobe mdio
modprobe ptp
modprobe dca
insmod /usr/src/netmap/LINUX/ixgbe/ixgbe.ko max_vfs=2,2
insmod /lib/modules/3.16.0-4-amd64/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko
And reload server!

 Then, I have physical NIC eth6 which represented as 2 virtual NICs (ip link show):
8: eth6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 90:e2:ba:4a:d8:e8 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 90:e2:ba:55:aa:bb, spoof checking on, link-state auto
    vf 1 MAC 90:e2:ba:55:bb:cc, spoof checking on, link-state auto
15: eth10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 90:e2:ba:55:aa:bb brd ff:ff:ff:ff:ff:ff
16: eth11: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 90:e2:ba:55:bb:cc brd ff:ff:ff:ff:ff:ff
Then, I configure MAC's for both virtual functions:
ip link set eth6 vf 0 mac 90:e2:ba:55:aa:bb
ip link set eth6 vf 1 mac 90:e2:ba:55:bb:cc
And reload ixgbevf driver (required for configuration):
rmmod ixgbevf
modprobe ixgbevf
After this operations all interfaces become ready:
ethtool eth6
Settings for eth6:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseT/Full
                            10000baseT/Full
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  1000baseT/Full
                            10000baseT/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Speed: 10000Mb/s
    Duplex: Full
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    Supports Wake-on: d
    Wake-on: d
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes
root@filteredclient ~ # ethtool eth10
Settings for eth10:
    Supported ports: [ ]
    Supported link modes:   10000baseT/Full
    Supported pause frame use: No
    Supports auto-negotiation: No
    Advertised link modes:  Not reported
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Speed: 10000Mb/s
    Duplex: Full
    Port: Other
    PHYAD: 0
    Transceiver: Unknown!
    Auto-negotiation: off
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes
root@filteredclient ~ # ethtool eth11
Settings for eth11:
    Supported ports: [ ]
    Supported link modes:   10000baseT/Full
    Supported pause frame use: No
    Supports auto-negotiation: No
    Advertised link modes:  Not reported
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Speed: 10000Mb/s
    Duplex: Full
    Port: Other
    PHYAD: 0
    Transceiver: Unknown!
    Auto-negotiation: off
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes
And run pkt-get example app on ixgbe VF:
./pkt-gen -f rx -i netmap:eth10
361.732713 main [1651] interface is netmap:eth10
361.733491 extract_ip_range [288] range is 10.0.0.1:0 to 10.0.0.1:0
361.733506 extract_ip_range [288] range is 10.1.0.1:0 to 10.1.0.1:0
361.734155 main [1848] mapped 334980KB at 0x7f5f9804f000
Receiving from netmap:eth10: 1 queues, 1 threads and 1 cpus.
361.734182 main [1934] Wait 2 secs for phy reset
363.734288 main [1936] Ready...
363.734328 nm_open [456] overriding ifname eth10 ringid 0x0 flags 0x1
363.734384 receiver_body [1184] reading from netmap:eth10 fd 4 main_fd 3
364.735415 main_thread [1448] 3 pps (3 pkts in 1001042 usec)
365.736481 main_thread [1448] 1 pps (1 pkts in 1001065 usec)
366.737542 main_thread [1448] 1 pps (1 pkts in 1001061 usec)
^C367.134692 main_thread [1448] 0 pps (0 pkts in 397151 usec)


No comments :

Post a Comment

Note: only a member of this blog may post a comment.