I think you're stuck with that unless you wire your own short "USB" to db-9 cable on the switch side.
I think brocade made some that were one piece so you'd eliminate the adapter but not fix the length.
Maybe find a USB to serial that's shorter? The real issue here is that nothing has serial ports anymore. Without the USB ->serial adapter your setup would be at worst marginally less convenient than a straight serial cable with db-9 on one end and rj45 on the other.
I think brocade made some that were one piece so you'd eliminate the adapter but not fix the length.
Maybe find a USB to serial that's shorter? The real issue here is that nothing has serial ports anymore. Without the USB ->serial adapter your setup would be at worst marginally less convenient than a straight serial cable with db-9 on one end and rj45 on the other.