Cisco VIC network card in standard PCI-E

[Takrbark3]

Cisco VIC network card in standard PCI-E

Hi!
I opened this topic to discuss the CISCO VIC series card using in non CISCO servers.
I have found of the these cards low price on ebay, so it will be perfect to build for 10G/40G/100G network.

These cards has Ethernet and FCoE interfaces and hardware-offload capabilities.

I have tested the following cards:
-CISCO VIC 1225: (Tested: WORKS)
-CISCO VIC 1285: (not yet tested)
-CISCO VIC 1385: (not yet tested)
-CISCO VIC 1455: (not yet tested)
-CISCO VIC 1495: (not yet tested)


CISCO VIC 1225

Details:
The card has PCI-E x16 slot, I tested in x8 mode too: worked.
The card has extra leg after the x16 side, doesn't need it, you can cover it (electrical isolation tape).

Connection:
The card support SM and MM SFP+ modules in 1G/10G mode.
I tested 2x SFP+ SM modules (FS branded): worked.

Windows drivers:
You need to download from the Cisco site:
Servers - Unified Computing > UCS C-Series Rack-Mount Standalone Server Software > xxx M4 > Unified Computing System (UCS) Drivers

Linux drivers:
You can use the built-in drivers or You can download from the Cisco site:
Servers - Unified Computing > UCS C-Series Rack-Mount Standalone Server Software > xxx M4 > Unified Computing System (UCS) Drivers

enic.ko: Ethernet driver for Linux.
fnic.ko: FCOE driver for Linux.

You dont need to load both of the modules, for ethernet is enough the "enic.ko":

modprobe enic

$> root@debian:~# modinfo enic
filename:       /lib/modules/4.19.0-6-amd64/kernel/drivers/net/ethernet/cisco/enic/enic.ko
version:        2.3.0.53
license:        GPL
author:         Scott Feldman <scofeldm@cisco.com>
description:    Cisco VIC Ethernet NIC Driver
srcversion:     7E57430A363BE1E7A2D3534
alias:          pci:v00001137d00000071sv*sd*bc*sc*i*
alias:          pci:v00001137d00000044sv*sd*bc*sc*i*
alias:          pci:v00001137d00000043sv*sd*bc*sc*i*
depends:       
retpoline:      Y
intree:         Y
name:           enic
vermagic:       4.19.0-6-amd64 SMP mod_unload modversions
sig_id:         PKCS#7
signer:         Debian Secure Boot CA
sig_key:        A7:46:8D:EF
sig_hashalgo:   sha256
signature:      7D:49:BB:9D:70:96:29:C3:53:3B:0D:D5:C8:F2:F0:5A:56:40:3C:C1:
       EF:7C:22:38:D0:4F:B4:57:8C:A1:64:5E:B3:5D:91:C6:64:C0:9D:49:
       74:78:74:32:4A:65:7A:A7:32:3A:4B:D8:67:71:CE:CE:C0:58:BB:30:
       68:06:6C:36:7C:66:5F:22:93:C1:A8:E0:7A:B4:9F:B7:4A:83:7F:C0:
       7C:28:2A:C0:96:76:63:5C:F7:D7:46:21:06:2A:59:AE:58:46:36:12:
       6D:1D:EE:82:FF:36:38:35:42:37:DA:40:30:8C:54:0A:2F:80:95:66:
       FE:AE:54:A8:EA:0C:9D:CB:9B:28:83:BC:27:40:8B:BA:DF:73:C4:3D:
       B3:54:71:58:C0:26:D6:A7:EB:D0:30:26:23:34:E3:5C:5A:E3:73:79:
       2E:D7:4E:E2:39:45:3A:8B:0C:BC:97:0D:74:74:BD:D8:CF:BA:59:84:
       B5:7C:1D:DF:E1:FC:0F:AD:A0:74:D4:E9:D7:29:3E:4A:33:13:2E:05:
       5D:E8:84:C2:FD:94:E8:1B:FE:02:E6:1A:5A:CB:32:31:DA:CA:C0:36:
       CA:8D:51:98:69:7F:5A:FB:25:5E:E1:97:FC:64:F8:A6:64:D6:D1:E1:
       64:B0:DC:76:0B:81:DE:CD:75:67:76:AD:27:D1:F8:62

$> root@debian:~# modinfo fnic
filename:       /lib/modules/4.19.0-6-amd64/kernel/drivers/scsi/fnic/fnic.ko
version:        1.6.0.34
license:        GPL v2
author:         Abhijeet Joglekar <abjoglek@cisco.com>, Joseph R. Eykholt <jeykholt@cisco.com>
description:    Cisco FCoE HBA Driver
srcversion:     AEFAB4CDE82CDF76F067C5A
alias:          pci:v00001137d00000045sv*sd*bc*sc*i*
depends:        scsi_mod,libfcoe,libfc,scsi_transport_fc
retpoline:      Y
intree:         Y
name:           fnic
vermagic:       4.19.0-6-amd64 SMP mod_unload modversions
sig_id:         PKCS#7
signer:         Debian Secure Boot CA
sig_key:        A7:46:8D:EF
sig_hashalgo:   sha256
signature:      2D:8F:A5:16:DE:24:8A:43:9F:86:C2:90:A8:D0:5B:B9:DC:53:A8:9A:
       25:6D:26:F0:63:E3:3C:E7:73:1C:D6:8F:A0:E0:D1:6A:6A:5D:74:67:
       BD:60:17:DE:10:4F:A1:52:BC:06:56:4C:F3:4B:CC:8E:C3:42:3F:91:
       13:99:B2:41:CF:82:1D:31:C3:9D:6C:90:68:06:78:71:6A:8F:77:09:
       E5:14:E3:2B:65:CE:45:F0:51:C8:99:DC:2F:1D:27:70:A9:FC:15:DF:
       B2:65:1C:92:6A:8F:5F:FF:C3:36:17:74:7A:DF:CF:F8:F5:34:6E:47:
       81:2D:B9:A2:FB:35:C6:CF:DB:69:5D:F9:55:C3:6F:DD:BE:91:CD:30:
       E2:CF:B4:CD:99:94:A1:C4:BE:78:19:4E:A6:B5:04:1F:B0:05:0A:2C:
       2D:E8:B9:D4:C4:6B:1C:11:C2:65:CC:7E:51:95:EC:C1:D1:DF:5C:62:
       5A:A7:15:C7:E0:9E:C6:54:8E:C7:7D:63:3A:39:5A:D2:7F:CA:41:FB:
       90:EE:6F:34:DF:15:AD:89:A9:38:7B:56:C3:98:69:19:28:16:EF:AC:
       31:B4:C5:95:09:B4:15:DC:38:92:B3:2A:06:EE:6C:A1:0A:DE:9B:D1:
       76:F9:5A:B3:74:74:73:28:40:F8:5A:53:6D:9D:9E:00
parm:           fnic_log_level:bit mask of fnic logging levels (int)
parm:           fnic_trace_max_pages:Total allocated memory pages for fnic trace buffer (uint)
parm:           fnic_fc_trace_max_pages:Total allocated memory pages for fc trace buffer (uint)
parm:           fnic_max_qdepth:Queue depth to report for each LUN (uint)

$> root@debian:~# ethtool -n enp9s0
       4 RX rings available
       Total 0 rules

$> root@debian:~# ethtool -g enp9s0
       Ring parameters for enp9s0:
       Pre-set maximums:
       RX:       4096
       RX Mini:   0
       RX Jumbo:   0
       TX:       4096
       Current hardware settings:
       RX:       512
       RX Mini:   0
       RX Jumbo:   0
       TX:       256

$> root@debian:~# ethtool -k enp9s0
       Features for enp9s0:
       rx-checksumming: on
       tx-checksumming: on
           tx-checksum-ipv4: off [fixed]
           tx-checksum-ip-generic: on
           tx-checksum-ipv6: off [fixed]
           tx-checksum-fcoe-crc: off [fixed]
           tx-checksum-sctp: off [fixed]
       scatter-gather: on
           tx-scatter-gather: on
           tx-scatter-gather-fraglist: off [fixed]
       tcp-segmentation-offload: on
           tx-tcp-segmentation: on
           tx-tcp-ecn-segmentation: on
           tx-tcp-mangleid-segmentation: off
           tx-tcp6-segmentation: on
       udp-fragmentation-offload: off
       generic-segmentation-offload: on
       generic-receive-offload: on
       large-receive-offload: off [fixed]
       rx-vlan-offload: on [fixed]
       tx-vlan-offload: on [fixed]
       ntuple-filters: off
       receive-hashing: on
       highdma: on [fixed]
       rx-vlan-filter: off [fixed]
       vlan-challenged: off [fixed]
       tx-lockless: off [fixed]
       netns-local: off [fixed]
       tx-gso-robust: off [fixed]
       tx-fcoe-segmentation: off [fixed]
       tx-gre-segmentation: off [fixed]
       tx-gre-csum-segmentation: off [fixed]
       tx-ipxip4-segmentation: off [fixed]
       tx-ipxip6-segmentation: off [fixed]
       tx-udp_tnl-segmentation: off [fixed]
       tx-udp_tnl-csum-segmentation: off [fixed]
       tx-gso-partial: off [fixed]
       tx-sctp-segmentation: off [fixed]
       tx-esp-segmentation: off [fixed]
       tx-udp-segmentation: off [fixed]
       fcoe-mtu: off [fixed]
       tx-nocache-copy: off
       loopback: off [fixed]
       rx-fcs: off [fixed]
       rx-all: off [fixed]
       tx-vlan-stag-hw-insert: off [fixed]
       rx-vlan-stag-hw-parse: off [fixed]
       rx-vlan-stag-filter: off [fixed]
       l2-fwd-offload: off [fixed]
       hw-tc-offload: off [fixed]
       esp-hw-offload: off [fixed]
       esp-tx-csum-hw-offload: off [fixed]
       rx-udp_tunnel-port-offload: on
       tls-hw-tx-offload: off [fixed]
       tls-hw-rx-offload: off [fixed]
       rx-gro-hw: off [fixed]
       tls-hw-record: off [fixed]

Hypervisor drivers:
You need to download from the Cisco site:
Servers - Unified Computing > UCS C-Series Rack-Mount Standalone Server Software > xxx M4 > Unified Computing System (UCS) Drivers

Supported: VMWARE, KVM, XEN, HYPERV

Other:
Additional cooling is required for production operation of the card, under the test the SFP+ module become hot ~70C - without cooling.

 

[oddball]

I have a bunch of these, all in UCS servers, the 10G and 40G ones, none of the 25/50/100 newer cards. Some in the mLom format, some PCI, and others in blades with the funky PCI connection.

I can see how they'd work if you install the drivers. But you're really missing most of what these cards are good for. They're virtual ethernet cards. You control and configure the NIC's via CIMC or UCS Manager. They have some really cool software failover as well, all transparent to the OS.

I don't really know how you can configure these without the management software. They are set with the worst settings out of the box, almost no queues, and weird config settings for buffers. I don't think SR-IOV is turned on by default either.

Glad you got it working though. I love the UCS series. They're solidly built servers, and they accept ANY PCI card and ANY drive. The VIC cards are really well built.

As for cooling, they should have heatsinks installed on them that in a 1U should keep them cool enough. If you have a UCS there is a temperature sensor on the card itself.

 

[Takrbark3]

I updated the 1st post.
You can change the card settings without UCS manager.
The VIC features/functions (Virtual Interfaces) will not work.

 

[plisc004]

Sorry if this is a bit of a necro, but I don't think this information is up-to-date anymore. The Windows drivers you reference are not recognized as compatible, and trying to manually use them results in an error where the device cannot start. Were there any additional steps needed to get this to work, or is there a specific version of the driver you used that may work?

 

[Takrbark3]

Sorry if this is a bit of a necro, but I don't think this information is up-to-date anymore. The Windows drivers you reference are not recognized as compatible, and trying to manually use them results in an error where the device cannot start. Were there any additional steps needed to get this to work, or is there a specific version of the driver you used that may work?

I tested the card under Linux, built in kernel driver worked out of the box ( no driver installation needed ).
The "fnic" is the FCOE driver (this is not need to operate the card, you can remove/blacklist this), "enic" is the ethernet driver.

I did test the card under Windows10, use the 4.0.4 series drivers:
"ucs-cxxx-drivers-windows.4.0.4d.iso"

 

[j43li]

Hi, I just bought two Cisco VIC network cards for my home lab. But I cannot download the drivers from Cisco's website. Can anyone kindly share the windows, linux and vmare drivers?

Thanks,

 

[oddball]

You should be able to register on Cisco's website for the downloads. None of the UCS stuff requires a support contract to download, just an account.

Each driver file is 4-6GB, they're bundles.

Which cards did you purchase?

 

[j43li]

Thanks for your reply.I bought UCSC-PCIE-CSC-02 10G Fiber Card.I didn't find the place to register of my UCS card.

 

[j43li]

I registered in Cisco website and downloaded ucs-cxxx-drivers-windows.4.3.1a.iso. It is free.

 

[Gqudi]

What transceivers did you all used. I tried HP branded ones. Bc I dont have cisco branded ones. Should I try Cisco transceivers.

 

[mrpasc]

You can use any CISCO transceiver or even the cheaper ones from fs.com or blueoptics as long as they are marked as Cisco compatible.