Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
I am on an EPYC 7601 so all PCI-E lanes are hanging off of CPU dies, however there can be similar issues to a PCI-E switch when running very high bandwidth devices. I am wondering if I should try software to see if I can pin the fio driver to some idle cores. Part of me wonders if it is the driver not being well optimized for odd numbers of cards. If I pull a card from the array and run two of them I lose about 1GB/s of write speed but read speed stays roughly the same.
Hmm, we obviously never had EPYC to develop the drivers against. No clue how interrupt handling is different vs the prior platforms.
There are ways in the driver to set flags for different types of interrupt scenarios, but there's no public facing docs on it. I'll have to comb my email history and see if there's anything in there.
You can use the fio utils to get/set a bunch of parameters that might alter performance one way or another.
I'd also bump your read threads in the test from 1 to 4 or more.
There are ways in the driver to set flags for different types of interrupt scenarios, but there's no public facing docs on it. I'll have to comb my email history and see if there's anything in there.
You can use the fio utils to get/set a bunch of parameters that might alter performance one way or another.
I'd also bump your read threads in the test from 1 to 4 or more.
These are for high bandwidth NICs but might contain useful information:
https://community.mellanox.com/s/article/how-to-tune-an-amd-server--eypc-cpu--for-maximum-performance
https://www.amd.com/system/files/TechDocs/56224.pdf
Bumping the Read threads to 4 made a big difference.
https://www.amd.com/system/files/TechDocs/56224.pdf
Bumping the Read threads to 4 made a big difference.
Just for S&Gs here is 64 threads (for all tests) on a 64 thread capable CPU striped across 3 IODrive2. Pretty good example of how Fusion IO devices scale well with CPU power.
Looking for a gut check before I go down the road of flashing my cards from eBay. I'd love to make them appear to my DL380p Gen8 as HP-branded cards so that the server can slow down its fans. Can I do this by mucking about with the .fff flash file and reflashing the card? I've got the latest drivers installed and working just fine under Ubuntu 16.04, and here's the output of fio-status:
Code:
Found 1 ioMemory device in this system
Driver version: 3.2.16 build 1731
Adapter: Single Controller Adapter
Fusion-io ioDrive2 1.205TB, Product Number:F00-001-1T20-CS-0001, SN:1213D1754, FIO SN:1213D1754
External Power: NOT connected
PCIe Power limit threshold: 24.75W
Connected ioMemory modules:
fct0: Product Number:F00-001-1T20-CS-0001, SN:1213D1754
fct0 Attached
ioDrive2 Adapter Controller, Product Number:F00-001-1T20-CS-0001, SN:1213D1754
Located in slot 0 Center of ioDrive2 Adapter Controller SN:1213D1754
PCI:04:00.0
Firmware v7.1.17, rev 116786 Public
1205.00 GBytes device size
Internal temperature: 41.83 degC, max 48.23 degC
Reserve space status: Healthy; Reserves: 100.00%, warn at 10.00%
Contained VSUs:
fioa: ID:0, UUID:80d5226e-14be-48cd-9adf-aa1ed6672176
fioa State: Online, Type: block device
ID:0, UUID:80d5226e-14be-48cd-9adf-aa1ed6672176
1205.00 GBytes device sizeHmm, I've never tried pushing the HP stuff onto an ioDrive. I've always gone the other direction.
There were bios updates for the HP servers that fixed the fan issues with the Fusion-io/SanDisk firmware.
I suppose I could tear into the HP firmware and look and see what stands out.
There were bios updates for the HP servers that fixed the fan issues with the Fusion-io/SanDisk firmware.
I suppose I could tear into the HP firmware and look and see what stands out.
Does the driver do IRQ steering?
If not, one option is to steer interrupts statically rather than letting the IRQ balancer change the steering dynamically.
This distributes the interrupts form the iodrive evenly across all the cores.
If you know which cores are preferred, you could specify the the 'cores' list.
Code:
IRQS=$(sed -n '/iodrive/s/:.*//p;' /proc/interrupts)
cores=($(seq 1 $(grep -c processor /proc/cpuinfo)))
i=0
for IRQ in $IRQS
do
core=${cores[$i]}
let "mask=2**(core-1)"
echo $(printf "%x" $mask) > /proc/irq/$IRQ/smp_affinity
let "i+=1"
if [[ $i ==${#cores[@]} ]]; then
i=0
fi
doneI think this might just be a sign that I should rip the E5-2697v2s out of this box and build a quieter Supermicro-backed desktop system. Thanks!
If anyone has any info about the male end of of the power connector that plugs in to a ioDrive 2 1.2TB, I'd appreciate it.
I plan to make a cable to connect it to a DL380 Gen8 PCI riser.
.
The hardware installation guide says the card draws up to 55W at 12V through this connector.
I plan to make a cable to connect it to a DL380 Gen8 PCI riser.
.The hardware installation guide says the card draws up to 55W at 12V through this connector.
Last edited:
That connector is unnecessary to plug in and is only for pci-e slots that cannot provide 25W of power.
That HP riser will provide 25w per slot no problem (assuming you plug in the HP power connector for it).
The 1.2TB cards won't use more than 25W on peak write.
Either way, if you want to build one, here's all the info you need Pins 1 and 2 on that 4-pin connector are 12v, the other two are ground.
Part numbers are included, and FrozenCPU is still in business. =)
That HP riser will provide 25w per slot no problem (assuming you plug in the HP power connector for it).
The 1.2TB cards won't use more than 25W on peak write.
Either way, if you want to build one, here's all the info you need Pins 1 and 2 on that 4-pin connector are 12v, the other two are ground.
Part numbers are included, and FrozenCPU is still in business. =)
Last edited:
Thanks so much @aquacow for the advice and details.
fio-status is reporting the 25W power limit. If that's sufficient maybe I should plan to do benchmarks first, and see if there's any issues before building power cables.
I'm not familiar with any optional extra cable to supply more power to the riser. I thought the white power connector on the riser was to supply power to supply additional power to a GPU. So I thought I'd be drawing power from that connector on the riser to power the iodrive.
fio-status is reporting the 25W power limit. If that's sufficient maybe I should plan to do benchmarks first, and see if there's any issues before building power cables.
I'm not familiar with any optional extra cable to supply more power to the riser. I thought the white power connector on the riser was to supply power to supply additional power to a GPU. So I thought I'd be drawing power from that connector on the riser to power the iodrive.
Aah, you're correct. Yeah, that's the HP GPGPU connector. There's power provided through the slot the riser plugs into...that's all you need for ioDrive IIs.
The 3TB and the ioDrive 3 can get up to 55W, so those are recommended for either 75W slots, or power connectors in older machines.
The 3TB and the ioDrive 3 can get up to 55W, so those are recommended for either 75W slots, or power connectors in older machines.
Hi all,
I just ordered myself this 1.2Tb card also, but not arrived yet.
I was just wondering where to get the drivers for windows7 or windows10?
Card ID is
F00-001-1T20-CS-0001
But does that tell if the card is branded for dell/hp etc?
I tried checking sandisk/wd support site but did not find the driver..
Thanks
I just ordered myself this 1.2Tb card also, but not arrived yet.
I was just wondering where to get the drivers for windows7 or windows10?
Card ID is
F00-001-1T20-CS-0001
But does that tell if the card is branded for dell/hp etc?
I tried checking sandisk/wd support site but did not find the driver..
Thanks
Here's a script that should install the drivers on Ubuntu 18.04.
Note: This only installs the drivers for the current kernel version. And you can't install two versions of the driver, due to conflicts in some of the files. So, after one installs a new kernel version, this requires rebuild for the new kernel version, removal of the previous driver version (due to conflicts), and install of the new driver version.
Code:
sudo apt-get install -y gcc fakeroot build-essential debhelper rsync
sudo apt-get install linux-headers-generic # ubuntu
sudo apt-get install -y libelf-dev
tar xzf iomemory-vsl_3.2.16.1731-1.0.tar.gz
cd iomemory-vsl-3.2.16.1731
cp ./root/usr/src/iomemory-vsl-3.2.16/kfio/.x86_64_cc63_libkfio.o.cmd ./root/usr/src/iomemory-vsl-3.2.16/kfio/.x86_64_cc73_libkfio.o.cmd
cp ./root/usr/src/iomemory-vsl-3.2.16/kfio/x86_64_cc63_libkfio.o_shipped ./root/usr/src/iomemory-vsl-3.2.16/kfio/x86_64_cc73_libkfio.o_shipped
patch -p0 <<EOF
--- fio-driver.spec.~1~ 2018-08-15 16:07:19.000000000 -0500
+++ fio-driver.spec 2019-03-03 08:00:54.949738642 -0600
@@ -328,8 +328,8 @@
/usr/src/iomemory-vsl-3.2.16/include/fio/port/linux/ktypes.h
/usr/src/iomemory-vsl-3.2.16/include/fio/port/linux/utypes.h
/usr/src/iomemory-vsl-3.2.16/include/fio/port/gcc/align.h
-/usr/src/iomemory-vsl-3.2.16/kfio/.x86_64_cc63_libkfio.o.cmd
-/usr/src/iomemory-vsl-3.2.16/kfio/x86_64_cc63_libkfio.o_shipped
+/usr/src/iomemory-vsl-3.2.16/kfio/.x86_64_cc73_libkfio.o.cmd
+/usr/src/iomemory-vsl-3.2.16/kfio/x86_64_cc73_libkfio.o_shipped
%changelog
--- ./debian/iomemory-vsl-source.install.~1~ 2018-08-15 16:54:59.000000000 -0500
+++ ./debian/iomemory-vsl-source.install 2019-03-03 08:46:38.258317811 -0600
@@ -119,6 +119,7 @@
usr/src/iomemory-vsl-3.2.16/kfio/.x86_64_cc54_libkfio.o.cmd
usr/src/iomemory-vsl-3.2.16/kfio/.x86_64_cc53_libkfio.o.cmd
usr/src/iomemory-vsl-3.2.16/kfio/.x86_64_cc63_libkfio.o.cmd
+usr/src/iomemory-vsl-3.2.16/kfio/.x86_64_cc73_libkfio.o.cmd
usr/src/iomemory-vsl-3.2.16/kfio/.x86_64_cc41_libkfio.o.cmd
usr/src/iomemory-vsl-3.2.16/kfio/.x86_64_cc44_libkfio.o.cmd
usr/src/iomemory-vsl-3.2.16/kfio/.x86_64_cc48_libkfio.o.cmd
@@ -128,5 +129,6 @@
usr/src/iomemory-vsl-3.2.16/kfio/x86_64_cc41_libkfio.o_shipped
usr/src/iomemory-vsl-3.2.16/kfio/x86_64_cc48_libkfio.o_shipped
usr/src/iomemory-vsl-3.2.16/kfio/x86_64_cc63_libkfio.o_shipped
+usr/src/iomemory-vsl-3.2.16/kfio/x86_64_cc73_libkfio.o_shipped
usr/src/iomemory-vsl-3.2.16/kfio/x86_64_cc44_libkfio.o_shipped
usr/src/iomemory-vsl-3.2.16/kfio/x86_64_cc49_libkfio.o_shipped
EOF
dpkg-buildpackage -b -uc -us
cd ..
apt remove -y iomemory-vsl-*-generic
dpkg -i iomemory-vsl-$(uname -r)_3.2.16.1731-1.0_amd64.deb
lsmod|grep iomem
sed '/^#/d;/^$/d' /etc/sysconfig/iomemory-vsl
sed -i 's/^#*ENABLED=1/ENABLED=1/' /etc/sysconfig/iomemory-vsl
systemctl restart iomemory-vsl
systemctl status -l iomemory-vsl
fio-status
Last edited: