Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
I tried the "dd temp-fanspeed" command with "?" and tab to see if FastIron had anything. I wasn't expecting anything, only curious. It didn't and instead I had to go physically reset the switch. The smart thing would have been to try on a switch I wasn't actively using, oh well.
really? what switch / fw version? That's one of the ones I did play with and had no issue, at least on 8030:

Code:
telnet@FCX2#dd temp-fanspeed
  DECIMAL   fan speed 1 or 2
  DECIMAL   gap between two readings (sec)
  auto      fan speed in auto mode
 
  • Like
Reactions: Emdee

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
Added the cheap FCX model to the main post as an option to learn BGP/VRFs/etc in a lab with - https://forums.servethehome.com/ind...s-cheap-powerful-10gbe-40gbe-switching.21107/

They predate the ICX line (came right before the ICX6610, in fact the ICX6610 and the FCX run the exact same bootloader and firmware images). They have no built in 10gbE, so you can get them cheap - around $30 for a BGP / VRF / tunnels / OSPF/ IPv6 / etc capable switch. The downside is no stock 10gbE as mentioned, they're a little louder, and draw a little more power, but not bad (around 40 watts, ICX6610 sound levels for the non PoE models, rocket ship sound levels for the PoE models). Overall I only recommend these if you're on a tight budget, don't really care about a lot of 10gbE, but want something capable of VRFs, BGP, OSPF, tunnels, etc to learn with

Advanced Routing License Note:
The FCXs required an EEPROM license to unlock the BGP and VRF capabilities. To do so, just run the following command from the bootloader (make sure to flash the latest bootloader from the guide first!):
Code:
i2c write a 0 feedface00000200ffffffffffffffff 1
reset
#will now boot and be permanently licensed for all advanced features

Notes on adding 10gbE to the FCX:

The FCX has no 10gbE by default. It comes with a card slot to add a 10gbE module. There's a 2 port 10gbE module, and a 4 port 10gbE module. There's two FCX switch types: one only takes the 2-port 10gbE card, and the other only takes the 4-port 10gbE card. To decide which you want/need:

First there's the "stacking" FCX models: These are the FCX624S and the FCX648S (the models with "S" at the end). These are by far the most popular and easiest to find for pennies. These have two special CX4 16gbps ports on the rear to stack with each other. These "stacking" models only take the 2-port 10gbE XFP card. Search ebay for "ES4625M" to find these cards around $40. This is the easiest option if you want a good lab switch with 2x 10gbE ports for around $70 total.

Then there's the more rare "datacenter" models, the FCX624-I, FCX624-E, FCX648-I and FCX648-E (24 or 48 port, intake or exhaust airflow direction). These do not have the special CX4 stacking ports on the back, and only take the 4-port SFP+ 10gbE card. The downside is, if you want to stack these models (they still support stacking), there's no dedicated CX4 ports in the back so you have to use one of the SFP+ ports for the stack connection. These models are harder to find, and typically more expensive. The SFP+ card for them is also more expensive. Search "FCX-4XG" on ebay to find the 4 port SFP+ cards for around $60. This means you can get an FCX datacenter model with 4 SFP+ ports for around $120 total, so it's not recommended over the quieter and more power efficient ICX6450 (unless you really need the BGP / VRFs / ETC and don't want to spring for the ICX6610 to get them)

Overall they're a little louder than the ICX6450 series, and predate the ICX line, but are incredibly powerful software wise and will do everything, so at $30 they can make great lab switches. For a permanent home network switch, or if you care about 10gbE, I would look at the newer models listed in the original post. I will say I've had a lot of these stuck in wiring closets and in use at LAN parties and they are rock solid models that will run forever, just missing out on a lot of 10gbE options
 
Last edited:

sean

Member
Sep 26, 2013
67
33
18
CT
really? what switch / fw version? That's one of the ones I did play with and had no issue, at least on 8030:

Code:
telnet@FCX2#dd temp-fanspeed
  DECIMAL   fan speed 1 or 2
  DECIMAL   gap between two readings (sec)
  auto      fan speed in auto mode
6450-24p with 8030s. I'll try again with a 48p tonight. Until the second attempt, it's hard for me to confirm exactly what happened. The scrollback looks right, but that only covers echoed characters.
 

eduncan911

The New James Dean
Jul 27, 2015
648
506
93
eduncan911.com
Anyone have a link to the 7250 serial cable? Searched the thread the eBay links are all old and not valid any longer.

My soldering station is down ATM and would rather just buy one.
 

klui

Well-Known Member
Feb 3, 2019
824
453
63
@fohdeesha or anyone else who has a 6610 with 2 Rev B/C PSUs.

Is the noise level lower with 2 PSUs vs 1 PSU? I have a 6610 with 2 fans and 1 Rev B PSU mounted in the far left slot when viewed from the rear and am curious if a second PSU will reduce the noise further. I find the noise lower with 2 fans vs. 1 fan and the temperatures are lower as well.


Thanks!
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
@fohdeesha or anyone else who has a 6610 with 2 Rev B/C PSUs.

Is the noise level lower with 2 PSUs vs 1 PSU? I have a 6610 with 2 fans and 1 Rev B PSU mounted in the far left slot when viewed from the rear and am curious if a second PSU will reduce the noise further. I find the noise lower with 2 fans vs. 1 fan and the temperatures are lower as well.


Thanks!
I went a step further with my 6610-24. (As historical data...I gutted my 6610-48, cut open the top of the chassis and put THREE 120mm fans on it. It was certainly low noise, wouldn't call it silent).

If you look at the 6610 fan trays, each tray has 2x fans with a bit of space between them, but they flow in the same direction. Not counter rotating so to speak, and not joined together. The fan connector for the tray has enough wires for the two fans. So...

Took two trays...took off one fan from each tray, wired the second tray to the first one (extend the wires enough to go around the stacking card) and put them back in. It is...subjectively quieter than two fan trays (with 4x fans vs 2x fans in two trays) with slightly less power draw.
 
  • Like
Reactions: fohdeesha

Ionitor

New Member
Jan 4, 2020
4
3
3
Speaking of the 7250 and serial communication...

I just got a 7250-24P off of eBay. I also purchased the DB9 female to USB mini console cable that vangoose linked previously. However, I cannot get any response from the switch (serial or otherwise). The power LED and ID #1 LEDs turn on, but nothing else. The fans spin high for a minute or so before slowing down, so it seems to be booting in some manner.

I've tried multiple serial ports at the other end, and I even double-checked the continuity on the cable pins. The only potential issue I can see from the cable is that it connects pins 1 and 4 on the mini USB end to the RTS/CTS flow control pins on the DB9 side, and everyone seems to agree that there is no hardware flow control on the 7250/7450 mini USB connectors. I wouldn't think that would prevent communication, though.

If I hook up a network cable to the administrative ethernet port, it doesn't connect. If I hook it up to one of the other ports, I get a connection, but nothing useful (within my limited experience troubleshooting switch connections). It doesn't pull an IP address from DHCP, and the Mikrotik switch it's attached to identifies it as an "edge" connection with no MAC addresses showing up in that connection.

Could this be a semi-dead switch, or is the cable the likely issue? Anyone had similar problems?

Edit: To eliminate the cable a bit more, I manually bridged only pins 2, 3, and 5 from the DB9 side of the cable to a serial port, bypassing the RTS/CTS pins. Still no communication.
 
Last edited:

KaHaR

New Member
Mar 1, 2017
9
2
3
39
Speaking of the 7250 and serial communication...

I just got a 7250-24P off of eBay. I also purchased the DB9 female to USB mini console cable that vangoose linked previously. However, I cannot get any response from the switch (serial or otherwise). The power LED and ID #1 LEDs turn on, but nothing else. The fans spin high for a minute or so before slowing down, so it seems to be booting in some manner.

I've tried multiple serial ports at the other end, and I even double-checked the continuity on the cable pins. The only potential issue I can see from the cable is that it connects pins 1 and 4 on the mini USB end to the RTS/CTS flow control pins on the DB9 side, and everyone seems to agree that there is no hardware flow control on the 7250/7450 mini USB connectors. I wouldn't think that would prevent communication, though.

If I hook up a network cable to the administrative ethernet port, it doesn't connect. If I hook it up to one of the other ports, I get a connection, but nothing useful (within my limited experience troubleshooting switch connections). It doesn't pull an IP address from DHCP, and the Mikrotik switch it's attached to identifies it as an "edge" connection with no MAC addresses showing up in that connection.

Could this be a semi-dead switch, or is the cable the likely issue? Anyone had similar problems?
I faced a similar issue with a wiped clean switch ... it's likely that the RJ45 Ethernet port has not been set up.
Plug in two devices to different ports and give them static IP addresses on the same subnet and see if they can talk to one another; if they can, the switch is probably fine.

I'm rather new to this switch, but for me, here's what I did ...
I purchased the cable I linked above and got a Tripplite/Keyspan serial to USB device and bought the "Serial" app from the App Store (this was the easiest way to get it working on my version of OSX as the install CD didn't install the right driver).

The first time I connected via Serial, it came right up, but subsequent attempts required a bunch of finagling and unplugging/replugging of both ends of the cable (to the switch/to the laptop's usb port) and sometimes the console wouldn't display the console without typing in a command first (e.g., it'd be a blank screen, but if you typed '?' it would display the help message). This may have been due to the computer falling asleep with the serial connection active, but I'm not certain.
 

dashpuppy

Member
Dec 16, 2018
48
21
8
Anyone know why the brocade switches (icx-6430-C12 ) would have issues displaying the web gui ? It's enable and have setup the user & aaa authentication login default local aaa authentication enable default local aaa authentication web default local commands but it only shows the port display. I can't get to any configuration menu's.

Bought a pair of them for Christmas, I have it up and running just having some little issues with vlans so i wanted to log into the GUI and see what i might be missing.

TIA !
 

Attachments

Ionitor

New Member
Jan 4, 2020
4
3
3
Speaking of the 7250 and serial communication...

I just got a 7250-24P off of eBay. I also purchased the DB9 female to USB mini console cable that vangoose linked previously. However, I cannot get any response from the switch (serial or otherwise). The power LED and ID #1 LEDs turn on, but nothing else. The fans spin high for a minute or so before slowing down, so it seems to be booting in some manner.

I've tried multiple serial ports at the other end, and I even double-checked the continuity on the cable pins. The only potential issue I can see from the cable is that it connects pins 1 and 4 on the mini USB end to the RTS/CTS flow control pins on the DB9 side, and everyone seems to agree that there is no hardware flow control on the 7250/7450 mini USB connectors. I wouldn't think that would prevent communication, though.

...

Edit: To eliminate the cable a bit more, I manually bridged only pins 2, 3, and 5 from the DB9 side of the cable to a serial port, bypassing the RTS/CTS pins. Still no communication.
Had a tickle of suspicion, so I tried reversing the send and receive pins (2 and 3) on my manual DB9 bridge and booted the switch again. I started getting text instantly!

So, as a heads up: do not buy the DB9 to Mini USB cable linked to by vangoose above (current listing here, sold by "tidunkin2012"). I'm going to work with the seller and I'll report back what they say.
 
  • Like
Reactions: fohdeesha

sash

Member
Nov 22, 2019
44
8
8
Need help understanding routing issue. I have bought ICX6610 off of eBay to replace Catalyst 3750G switch. I have configured ICX to be a layer 3 switch with OSPF routing between it the the router. Everything works as expected, except the fact that ICX itself cannot access the internet. I can ping internal clients as well as VPN subnets without issues but not the internet hosts. All other local clients have no issues accessing anything at all they are configured to access.

interface ethernet 1/3/1
port-name Uplink to EdgeRoute6 on port eth5
ip address 192.168.29.1 255.255.255.0
ip ospf area 0


SSH@ICX6610-48P#sh ip route
Total number of IP routes: 17
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
Destination Gateway Port Cost Type Uptime
1 0.0.0.0/0 192.168.29.254 e 1/3/1 1/1 S 2d23h
2 10.0.0.0/8 DIRECT ve 100 0/0 D 3d2h
3 172.16.31.0/30 192.168.29.254 e 1/3/1 110/11 O 2d23h
4 172.16.32.0/30 192.168.29.254 e 1/3/1 110/11 O 2d23h
5 172.16.33.0/30 192.168.29.254 e 1/3/1 110/11 O 2d23h
6 172.16.34.0/30 192.168.29.254 e 1/3/1 110/11 O 2d23h
7 172.16.35.0/30 192.168.29.254 e 1/3/1 110/11 O 2d23h
8 192.168.0.0/24 DIRECT ve 2 0/0 D 2d23h
9 192.168.1.0/24 192.168.29.254 e 1/3/1 110/21 O 2d23h
10 192.168.3.0/24 DIRECT ve 3 0/0 D 3d2h
11 192.168.7.0/24 192.168.29.254 e 1/3/1 110/12 O 2d23h
12 192.168.11.0/24 DIRECT ve 11 0/0 D 3d2h
13 192.168.12.0/24 DIRECT ve 12 0/0 D 3d2h
14 192.168.23.0/24 192.168.29.254 e 1/3/1 110/21 O 2d23h
15 192.168.29.0/24 DIRECT e 1/3/1 0/0 D 2d23h
16 192.168.35.0/24 DIRECT ve 35 0/0 D 3d2h
17 192.168.254.0/24 DIRECT ve 254 0/0 D 3d2h

SSH@ICX6610-48P#ping 8.8.8.8
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.

SSH@ICX6610-48P#traceroute 8.8.8.8

Type Control-c to abort
Tracing the route to IP node 8.8.8.8(8.8.8.8) from 1 to 30 hops

1 <1 ms <1 ms <1 ms er-6p [192.168.29.254]
2 * * * ?
3 * * * ?
4 * * * ?
5 * * * ?
6 * ^C
Trace Route aborted!
 
Last edited:

infoMatt

Active Member
Apr 16, 2019
222
100
43
Need help understanding routing issue. I have bought ICX6610 off of eBay to replace Catalyst 3750G switch. I have configured ICX to be a layer 3 switch with OSPF routing between it the the router. Everything works as expected, except the fact that ICX itself cannot access the internet. I can ping internal clients as well as VPN subnets without issues but not the internet hosts. All other local clients have no issues accessing anything at all they are configured to access.
[...]
Looking at the switch side, it has the correct routing table and on a traceroute the first hop is your edgerouter... Are you sure that the configuration on the ER is all right? Firewall rules, NAT/Masquerading,...?

EDIT: Also, keep in mind that the switch sends the packets with (if I remember correctly) the address of the lowest numbered loopback interface or (in case there are no loopbacks), the lowest VE; in the ER, you need to allow and properly route/masquerade the traffic from that source address...
 
Last edited:

sash

Member
Nov 22, 2019
44
8
8
Looking at the switch side, it has the correct routing table and on a traceroute the first hop is your edgerouter... Are you sure that the configuration on the ER is all right? Firewall rules, NAT/Masquerading,...?

EDIT: Also, keep in mind that the switch sends the packets with (if I remember correctly) the address of the lowest numbered loopback interface or (in case there are no loopbacks), the lowest VE; in the ER, you need to allow and properly route/masquerade the traffic from that source address...
Thank you for the response. I am not using ve on the Brocade side. I have issued enable command on the eth 1/3/1, assigned in an IP address and enabled OSPF on it. I figured that it is easier than messing with vlans and virtual interfaces. On the Edgerouter side I have not done any changes what so ever. It looks like the Edge router does not have the route back to Brocade for traffic destined for the switch. It is a point to point connection. So I am at a loss here. Not that I really need the switch to access the internet directly, but having the time sync would be good.

P.S. I have not configured loop back interface on the Brocade. Lowest VE interface would be ve 2. I'm not using vlan 1 at all. There is OSFP running on ve 2 interface. So it should be routing correctly...

There are all the VEs I have configured:

interface ve 2
ip address 192.168.0.11 255.255.255.0
ip ospf area 0
!
interface ve 3
ip address 192.168.3.1 255.255.255.0
ip helper-address 1 192.168.35.4
ip ospf area 0
!
interface ve 11
ip address 192.168.11.1 255.255.255.0
!
interface ve 12
ip address 192.168.12.1 255.255.255.0
ip helper-address 1 192.168.35.4
ip ospf area 0
!
interface ve 35
ip address 192.168.35.1 255.255.255.0
ip ospf area 0
!
interface ve 100
ip address 10.0.0.1 255.0.0.0
!
interface ve 254
ip address 192.168.254.1 255.255.255.0
 
Last edited:

infoMatt

Active Member
Apr 16, 2019
222
100
43
Thank you for the response. I am not using ve on the Brocade side. I have issued enable command on the eth 1/3/1, assigned in an IP address and enabled OSPF on it. I figured that it is easier than messing with vlans and virtual interfaces. On the Edgerouter side I have not done any changes what so ever. It looks like the Edge router does not have the route back to Brocade for traffic destined for the switch. It is a point to point connection. So I am at a loss here. Not that I really need the switch to access the internet directly, but having the time sync would be good.

P.S. I have not configured loop back interface on the Brocade. Lowest VE interface would be ve 2. I'm not using vlan 1 at all. There is OSFP running on ve 2 interface. So it should be routing correctly...

There are all the VEs I have configured:

interface ve 2
ip address 192.168.0.11 255.255.255.0
ip ospf area 0
!
[..]
Well, you have more than one VE, so you're effectively using those :p
Ensure on the edgerouter that it knows where to route the packets for 192.168.0.11/24 via 192.168.29.1.
But please ensure that the multiple IPs that you've given to the switch belogs to different VLANs, otherwise some nasty things might occour...
Also, if you're lost, some PCAP is always useful :p
 

Hakujou

New Member
Apr 28, 2019
20
7
3
Hello,

Did anyone managed to get VRF working on ICX6610 ?
When I create one and try to enable address-family on it, it complains with:
Code:
SSH@sw-core-1(config-vrf-secure)#address-family ipv4
Error: has reached maximum system limit of maximum number of IPv4 routes
       available IPv4 routes for non-default VRF 0
Which is weird, because the limit of routes is set at 1024 ipv4/100 ipv6 and I have no other VRF or significant routes configured on it.
Code:
SSH@sw-core-1(config-vrf-SECURE)#show default values
sys log buffers:50         mac age time:300 sec       telnet sessions:5

ip arp age:10 min          bootp relay max hops:4     ip ttl:64 hops
ip addr per intf:24

when multicast enabled :
igmp group memb.:260 sec   igmp query:125 sec         hardware drop: enabled

when ospf enabled :
ospf dead:40 sec           ospf hello:10 sec          ospf retrans:5 sec
ospf transit delay:1 sec

when bgp enabled :
bgp local pref.:100        bgp keep alive:60 sec      bgp hold:180 sec
bgp metric:10              bgp local as:1             bgp cluster id:0
bgp ext. distance:20       bgp int. distance:200      bgp local distance:200

System Parameters    Default    Maximum    Current    Configured
ip-arp               4000       64000      4000       4000
ip-static-arp        512        6000       512        512
ip-cache             10000      32768      10000      10000
ip-filter-port       3066       3066       3066       3066
ip-filter-sys        2048       8192       2048       2048
l3-vlan              32         1024       32         32
ip-qos-session       1024       16000      1024       1024
mac                  32768      32768      32768      32768
ip-route             12000      15168      12000      12000
ip-static-route      64         2048       64         64
vlan                 64         4095       64         64
spanning-tree        32         254        32         32
mac-filter-port      16         256        16         16
mac-filter-sys       32         512        32         32
ip-subnet-port       24         128        24         24
session-limit        8192       16384      8192       8192
view                 10         65535      10         10
virtual-interface    255        512        255        255
hw-traffic-condition 896        896        896        896
rmon-entries         1024       32768      1024       1024
igmp-snoop-mcache    512        8192       512        512
mld-snoop-mcache     512        8192       512        512
ip6-route            908        2884       908        908
ip6-static-route     178        576        181        181
ip6-cache            908        2884       908        908
msdp-sa-cache        4096       8192       4096       4096
gre-tunnels          16         64         16         16
hw-ip-route-tcam     16384      16384      16384      16384
ip-vrf               16         16         16         16
ip-route-default-vrf 12000      15168      12000      12000
ip6-route-default-vr 908        2884       908        908
ip-route-vrf         1024       15168      1024       1024
ip6-route-vrf        100        2884       100        100
pim-hw-mcache        1024       6144       1024       1024
pim6-hw-mcache       512        1024       512        512
igmp-snoop-group-add 4096       8192       4096       4096
mld-snoop-group-addr 4096       8192       4096       4096
mac-notification-buf 4000       16000      4000       4000
traffic-policies-sys 1024       1024       1024       1024
dot1x-mka-policy-gro 8          8          8          8
openflow-flow-entrie 1024       12000      1024       1024
openflow-pvlan-entri 40         256        40         40
openflow-unprotected 40         256        40         40
openflow-group-selec 0          120        0          0
openflow-nexthop-ent 0          1024       0          0
max-dhcp-snoop-entri 1024       3072       1024       1024
max-static-inspect-a 512        1024       512        512
Is it a bug or am I missing something here ?

Thanks

EDIT: Figured it out. ip-route-default-vrf was set at the value of ip-route, making no left route availables for non-default vrf. Lowered it allowed vrf to enable address-family.
 
Last edited:

Ionitor

New Member
Jan 4, 2020
4
3
3
So, as a heads up: do not buy the DB9 to Mini USB cable linked to by vangoose above (current listing here, sold by "tidunkin2012"). I'm going to work with the seller and I'll report back what they say.
The seller said that they had no idea whether the reversed pins was an error or if that cable is not intended to work with the ICX 7250/7450. So, sounds like it's not the right option. I'm either going to make my own cable or order the double-adapter others linked to.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,076
113
33
fohdeesha.com
Hello,

Did anyone managed to get VRF working on ICX6610 ?
When I create one and try to enable address-family on it, it complains with:
Code:
SSH@sw-core-1(config-vrf-secure)#address-family ipv4
Error: has reached maximum system limit of maximum number of IPv4 routes
       available IPv4 routes for non-default VRF 0
Which is weird, because the limit of routes is set at 1024 ipv4/100 ipv6 and I have no other VRF or significant routes configured on it.
Code:
SSH@sw-core-1(config-vrf-SECURE)#show default values
sys log buffers:50         mac age time:300 sec       telnet sessions:5

ip arp age:10 min          bootp relay max hops:4     ip ttl:64 hops
ip addr per intf:24

when multicast enabled :
igmp group memb.:260 sec   igmp query:125 sec         hardware drop: enabled

when ospf enabled :
ospf dead:40 sec           ospf hello:10 sec          ospf retrans:5 sec
ospf transit delay:1 sec

when bgp enabled :
bgp local pref.:100        bgp keep alive:60 sec      bgp hold:180 sec
bgp metric:10              bgp local as:1             bgp cluster id:0
bgp ext. distance:20       bgp int. distance:200      bgp local distance:200

System Parameters    Default    Maximum    Current    Configured
ip-arp               4000       64000      4000       4000
ip-static-arp        512        6000       512        512
ip-cache             10000      32768      10000      10000
ip-filter-port       3066       3066       3066       3066
ip-filter-sys        2048       8192       2048       2048
l3-vlan              32         1024       32         32
ip-qos-session       1024       16000      1024       1024
mac                  32768      32768      32768      32768
ip-route             12000      15168      12000      12000
ip-static-route      64         2048       64         64
vlan                 64         4095       64         64
spanning-tree        32         254        32         32
mac-filter-port      16         256        16         16
mac-filter-sys       32         512        32         32
ip-subnet-port       24         128        24         24
session-limit        8192       16384      8192       8192
view                 10         65535      10         10
virtual-interface    255        512        255        255
hw-traffic-condition 896        896        896        896
rmon-entries         1024       32768      1024       1024
igmp-snoop-mcache    512        8192       512        512
mld-snoop-mcache     512        8192       512        512
ip6-route            908        2884       908        908
ip6-static-route     178        576        181        181
ip6-cache            908        2884       908        908
msdp-sa-cache        4096       8192       4096       4096
gre-tunnels          16         64         16         16
hw-ip-route-tcam     16384      16384      16384      16384
ip-vrf               16         16         16         16
ip-route-default-vrf 12000      15168      12000      12000
ip6-route-default-vr 908        2884       908        908
ip-route-vrf         1024       15168      1024       1024
ip6-route-vrf        100        2884       100        100
pim-hw-mcache        1024       6144       1024       1024
pim6-hw-mcache       512        1024       512        512
igmp-snoop-group-add 4096       8192       4096       4096
mld-snoop-group-addr 4096       8192       4096       4096
mac-notification-buf 4000       16000      4000       4000
traffic-policies-sys 1024       1024       1024       1024
dot1x-mka-policy-gro 8          8          8          8
openflow-flow-entrie 1024       12000      1024       1024
openflow-pvlan-entri 40         256        40         40
openflow-unprotected 40         256        40         40
openflow-group-selec 0          120        0          0
openflow-nexthop-ent 0          1024       0          0
max-dhcp-snoop-entri 1024       3072       1024       1024
max-static-inspect-a 512        1024       512        512
Is it a bug or am I missing something here ?

Thanks

EDIT: Figured it out. ip-route-default-vrf was set at the value of ip-route, making no left route availables for non-default vrf. Lowered it allowed vrf to enable address-family.
yep, as the very first page of the VRF configuration section of the manual states, you have to lower the amount of routes assigned to the default VRF so you have some to assign to non-defaults. I typically just run:

system-max ip-route-default-vrf 9000
system-max ip-route-vrf 128