ESXI TrueNAS Core 13 VM, at my whit's end

[GW2]

Hi all, I will try to keep this brief so you don't get bored but get all the info you need in here

Hardware
Server: Dell T410
CPU: 2x E5-2450 v2 (8C 16T 2.5GHz)
Memory: 192GB DDR3 1600 MHz (12 x 16GB)
RAID Controller: Dell PERC H710P (used for VM Storage, truenas OS drive is stored here)
Virtual Disk 1: RAID5 8 x 1TB SATA SSD (temporary until I can get the truenas zfs storage working)
SAS HBA 1: Dell 6Gbps SAS HBA (External H310) flashed to IT mode (Passed through PCIE to truenas VM)
SAS HBA 2: Dell H310 Flashed to IT mode Passed through PCIE to truenas VM) (bought this as my SAS HBA was not working)
SAS shelf: Dell MD 1000 15 bay 3.5" SAS enclosure
12 x 8TB Seagate IronWolf drives
NIC: Asus XG-C100F
OS: ESXi 6.7

Truenas VM
BSD12 or newer setting
2 vCPU
12GB RAM
virtual nic
SAS HBA or H310 passed through on PCIE

The HBAs are both crossflashed to LSI 2008 it mode, and so far as I can tell everything went well with that and I had no issues flashing them, but I am prepared to accept these cards could be the cause of me problem

Truenas installs fine, I can see my disks, I can make a pool, a dataset, share it via smb, and connect to this share without issue.
I can copy a small file to the share and it will take a long time but it will copy, and be very fast to copy back.
I cannot copy multiple files or large files to the server, I get "Error 0x8007003B: An unexpected network error occurred."
I have tried disabling hardware offloading for networking on truenas to see if that is possible an issue, but it did not seem to help, tried a million different things at this point and have reinstalled at least 10 times the last 2 weeks without ever having anything even resembling success. I don't even need it to be particularly performant I just need it to be usable, I wanted to move from RAID to ZFS to keep my data safer, but for some reason I just cannot get truenas core to work for the life of me.

Any and all help is greatly appreciated

Thanks,
GW2

 

[Rand__]

U tried TNS instead? Just for testers...
Orher vms work fine i assume?
Nothing in the logs?

 

[GW2]

I have run ESXI on this server for years, never had an issue with any other VM but I'm all windows, so not used to other things so much. I had this H710 passed through to my plex server because it was not compatible with ESXI, but I needed it for the shelf to connect. which was originally 15x 3TB RAID6. I want to move to ZFS to keep my data a bit safer I got some 8TB drives to upgrade the array at the same time and so I don't have to worry about my data being destroyed if I mess it up.

I have not tried Scale, I understand core is supposed to be more performant, but I suppose if it does not perform at all there is no harm in trying scale.

I do have an intel NIC arriving soon that I want to try to remove the NIC as a variable as I'm using that Asus Aquantia based NIC, but if that does not help I will have to try scale

 

[coolrunnings82]

Do you have more than one network interface or vlans attached to TrueNas? Are you accessing it across another VLAN or from outside the subnet it's on?

 

[GW2]

No VLANS at all, flat 192.168.1.1/24 network, all systems on the same switch, only a virtual NIC is passed though, does the TrueNAS Core NIC need to be physical?

 

[Rand__]

No, vmxnet3 used to work just fine - at least in the last one i ran virtually. Also, if u can see it it in interfaces it should work.

Maybe Network is just an effect and suffers from other issues? If even small files are slow then large files just might not complete (chunked write of large blocks) and cause a timeout that gets sent back to network?

-Have you tried copying via a different protocol (ssh/scp from windows, nfs from linux)?
-Have you tried copying via a client OS (Linux VM, another freeBSD installation)?
-Have you shared the zfs pool drives over both haba's or just one? does either have the problem?
-Is it on the SAS enclosure only or also on locally attached disks?
-Have you tried giving the box more memory? 12 is very low (might even be below recommended minimum), a ZFS filer need more to perform well (memory=primary write cache)....

 

[GW2]

1. have not tried ssh/scp, i do not use linux, or ssh, or scp or nfs for anything in my environment, it's a home lab
2. Don't have or use any linux or freebsd anywhere else, just trying to setup truenas core
3. I made a new ZFS pool using the same drives on both HBAs (one at a time), on separate installs of truenas, no difference no matter the HBA
4. only a SAS enclosure and 1 virtual disk for the OS
5. I had it configured with 64GB of RAM and made no difference at all, truenas forum told me i am assigning too many resources which can cause problems which is why i brought it back down for testing, RAM usage never goes beyond less than 1GB in truenas, so don't think it's RAM related at all, it should still work even if it had low ram though really right?

I have time this weekend finally to look into it again, going to try switching to try passing an intel NIC through, and then try it on the host. I do kinda suspect it may be that I'm not using an Intel NIC so something isn't handled quite right. If those don't work I will have to try a different model HBA, will probably just get an LSI card instead of trying the cross-flashed cards. I will update in a bit, I may fiddle with setting up a linux VM but I don't really want to as it's not pertinent to the environment it will be used in, but I guess it may help pinpoint the problem so I may have to suck it up. I think my other final option is try TrueNAS Scale as it's linux based and has better hardware compatibility but I chose core because it's supposed to perform a little better and I already have ESXI for the virtualization aspect that scale offers.

One more question I had for anyone who may know but are there any BIOS settings that can affect things like this on a Dell server?

Thanks so much for your replies, appreciate any input at all.

 

[GW2]

Okay, so the issue the whole time was indeed the NIC. I also changed it to boot BIOS instead of UEFI but I am not convinced that had anything
to do with it tbh, I will reinstall and test on UEFI now as I think it's knowing. (I did it and it works on UEFI too so it was the NIC 100%)

Aquantia NIC -> Virtual switch -> vmxnet3 NIC -> TrueNAS Core - does not work
Intel NIC -> PCIE Passthrough to TrueNAS Core - works
Intel NIC -> Virtual switch -> vmxnet3 NIC -> TrueNAS Core - works

 

[Rand__]

Glad you found it

 

[GW2]

Sorry to bump this again but I was getting slowdown after prolonged file transfer down to 10MB/sec over a 5 gig link from my workstation, I went back to passthrough as this is the only machine that needs 10G anyway, and its working flawlessly for many hours of copying now, ranging in speeds from 90-350+MB/sec.

 

[umnum]

Hi, so I am having a similar issue and I tried everything but still having issues.

I have TrueNas Scale virtualized on an esxi host. I have a pfsense firewall between VLANs. TrueNAS is on VLAN 100 and dekstops are on VLAN 102. I can set up the SMB share on both linux and windows machines, copy most of yhe files without issues and with good speeds but there are couple files that always fail. When I move the TrueNAS to the same VLAN (102) with the desktops, the problem files copy without issues.

I switched the nic on the host, tried every setting in pfsense but I am still getting the network error when trying to copy the same problem files. Other files are copying just fine.

Can someone help me with this?

 

[GW2]

I didn't even fully solve it in the end, I ended up just installing Truenas Core on the hardware directly and now have no problems, but I understand some people will not want to do that, for me it's just a home server, so was really not a big deal, and I only had 1 other VM because truenas absorbed most of my other VMs functionality (unifi controller, plex server, etc)

 

[umnum]

This is driving me crazy. I don't get how just some certain files fail with a network error, it doesn't make sense. I tried a passthrough intel NIC to TrueNAS vm but still the same result. And still when I put the TrueNAS in the same VLAN everything works. But between VLANS most files work, some certain ones start to copy but fail in the middle somewhere. It is always the same files. Also I can copy those same files to anywhere else as well without issues.

 

[GW2]

Can I ask what NIC are you using physical I found that TrueNAS really seems to care if your NIC is not intel, even if its virtualized.

 

[umnum]

I have an Intel I350 PCIe card which I tested with passthrough. Also on board dual ethernet ports of the Supermicro X10DRH-IT mobo

 

[umnum]

There is always layer 3 connectivity between TrueNAS and the desktop computers so I don't think it is a routing issue. Most people solved it by fixing a dual route issue in their network but I don't have that. Only some specific files fail, everything else is just working. Those video files have a little bit higher bit rate than others, but not much. Also the failing files are not big compared to others.

 

[GW2]

What version of ESXi btw?

 

[umnum]

7U3

 

[GW2]

I seemed to have more luck 6.7, once I went to 7 it was horribly unstable with TrueNAS. TrueNAS is kind of hard to troubleshoot because documentation is lacking and it has a small approved hardware list and requires knowledge of BSD/Unix to troubleshoot at all. It is built to sell their appliances, so it works like a charm there, but everywhere else you are on your own I think, the TrueNAS forum was never very much help except to tell you to buy newer hardware, my hardware is kinda old but it's ivy bridge, which is new enough for good pcie passthrough and its a real server so it's got all the features ESXi could dream of like ECC, redundant PSUs, internal SAS backplane etc. and again everything works fine on hardware. I dunno man, that's why I gave up. You can a VM or two in truenas core running on hardware without issue, I run my only remaining VM on it. Or you can try virtualizing TrueNAS Scale as it runs on linux and is more friendly to leftover hardware style builds

 

[umnum]

I tried both core and scale, same results. The logs in core had more detail but I couldn't find anything online when I searched it online. IT was giving me stat_ex.st_ex_calculated_birthtime is different than expected error. I don't know what to do, this doesn't make sense at all.

I copied a bunch of stuff without any issues just by changing the port group of the TrueNAS to be on the same VLAN with the desktops. But as soon as I change it to be on another subnet, the same files keep failing to copy.