Never mind, I'm an idiot. just figured it out, just remembered that I could use the vif IPs to access the brocade switch on any of my VLANs. Same thing here, realized that i just needed to use the corresponding pfsense interface IP on the TRANSIT network as the pfsense GUI!
All fixed :)