Intel Max 1550 ES (QZFN) — 128GB HBM2e on the desktop: feasibility, oneAPI/llama.cpp backend status, carrier board options

[Raineypenguin]

The Intel Max 1550 ES (QZFN) units floating around eBay and all the usual suspects have caught my attention for an unusual reason: 128GB HBM2e at 3.2 TB/s is a genuinely interesting memory bandwidth figure for local LLM inference, and the ES units are sitting around $4k which while not cheap, is in the same conversation as an A100 80GB PCIe.

The obvious problem is software. CUDA and ROCm are established. oneAPI/SYCL exists, and llama.cpp has an Intel GPU backend (SYCL path), but I haven’t found much documentation on anyone actually running it on these data center gpus specifically, definitely not in the at home space. If anyone has tested the llama.cpp SYCL backend on Max series hardware I’d be interested in how painful the setup actually was and what performance looked like!

The second problem is getting the thing connected to a desktop system. Max 1550 is OAM, but the seller has a carrier board as well. They said they got it recognized in device manager on windows, but had no drivers.

Not expecting this to be a plug-and-play situation. More curious whether it’s possible in a non professional environment. If I would need Intel internal tools for setting up such an unsupported instance then I’d be square out of luck.

 

[charlatanshost]

I've been looking at the same card you've seen. The main problem that I haven't been able to find out is if the es require separate unreleased drivers, or the already available drives work.
and if it actually needs a vendor verified backplane board to actually run.
But the Intel Max series cards have really interest me, especially how opensource the software is.
I've been considering intel max 1100 cards for a month now, and waiting on more 1550 cards to hit the market at a price point I can afford to mess around with one, and hopefully production and not ES.

 

[Raineypenguin]

I've been looking at the same card you've seen. The main problem that I haven't been able to find out is if the es require separate unreleased drivers, or the already available drives work.
and if it actually needs a vendor verified backplane board to actually run.
But the Intel Max series cards have really interest me, especially how opensource the software is.
I've been considering intel max 1100 cards for a month now, and waiting on more 1550 cards to hit the market at a price point I can afford to mess around with one, and hopefully production and not ES.

Yeah it being an ES isn’t ideal, but it’s the first 1550 I’ve seen pop up. I think the seller having the OAM to pcie adapter board is the real find.

 

[charlatanshost]

The oam adapter is the same Chinese one that is used for the a100 OAM from what he told me.
So the one from Jungle J, same guy on Reddit, will work with it, if we can confirm that it isn't reliant on the vendor verification.
But that can't be confirmed without the drivers, I should ask if he's tried the regular Intel max drivers.

 

[Patriot]

The oam adapter is the same Chinese one that is used for the a100 OAM from what he told me.
So the one from Jungle J, same guy on Reddit, will work with it, if we can confirm that it isn't reliant on the vendor verification.
But that can't be confirmed without the drivers, I should ask if he's tried the regular Intel max drivers.

A100 doesnt use oam

 

[charlatanshost]

Sorry my mistake, you're correct. But for some weird reason that is what they call the adapter boards, a100/h100/h800 oam to pcie adapters.
I guess it has to do with just converting one of the sxm4 or 5 boards to oam socket? I'm not really sure, I've messaged the guy that makes them a few days ago for more information but he hasn't replied again yet. Last time I spoke to him was last year when I was looking at getting an sxm4 adapter, because he made one with nvlink.

 

[Patriot]

Sorry my mistake, you're correct. But for some weird reason that is what they call the adapter boards, a100/h100/h800 oam to pcie adapters.
I guess it has to do with just converting one of the sxm4 or 5 boards to oam socket? I'm not really sure, I've messaged the guy that makes them a few days ago for more information but he hasn't replied again yet. Last time I spoke to him was last year when I was looking at getting an sxm4 adapter, because he made one with nvlink.

Yeah its more their mistake, just making sure you dont try to connect a OAM accelerator onto a SXM adapter.
I have seen the listing before a few years ago but now the SXM adapters are more readily available and cheaper but I have yet to find a OAM one.

I would be interested in one tho lol....in addition to 2 hives of pcie mi100s, I have an OAM one.

 

[charlatanshost]

Yeah its more their mistake, just making sure you dont try to connect a OAM accelerator onto a SXM adapter.
I have seen the listing before a few years ago but now the SXM adapters are more readily available and cheaper but I have yet to find a OAM one.

I would be interested in one tho lol....in addition to 2 hives of pcie mi100s, I have an OAM one.

Oh yeah, no I appreciate that. Definitely wouldn't want to do that.
If the guy that builds them messages me back, I'll let you know, I've been communicating with him over reddit, he has been active in the past month, posted about his nvlink board.
But the oam mi100 sounds like a sweet card. I've honestly been kind of leary of most OAM cards mainly because of lack of information, or success, like the mi250x lack of successful attempts, and the Intel gaudi series lack of information.
Also I read some where the mi250x had vendor locks? But I can't remember where I read that, so that turned me away from them really fast.
But I'm still very interested in the Intel max 1550 if I can get my hands on a production model, might have to contact my local university as apparently they had/have some.

 

[Civiloid]

1550 ES was available on chinese platforms for a while, but rumours says that ES should be avoided as they are unreliable and dies rather quickly if you'll manage to get them working. Those are just rumours though, but enough for me not to risk my money.

 

[charlatanshost]

1550 ES was available on chinese platforms for a while, but rumours says that ES should be avoided as they are unreliable and dies rather quickly if you'll manage to get them working. Those are just rumours though, but enough for me not to risk my money.

That's a possibility, it is a 600watt tdp card after all.
Which does make me feel better about my 1100 I just received

 

[Civiloid]

That's a possibility, it is a 600watt tdp card after all.

I heard this about those ES that are available specifically, that it whatever the problem was - it got fixed in later stages and retail cards are fine from that perspective.

 

[charlatanshost]

I heard this about those ES that are available specifically, that it whatever the problem was - it got fixed in later stages and retail cards are fine from that perspective.

Ah gotcha, yeah the es 1550 is the only one I've seen so far.
Question, where are you able to find this information? Because I've been looking everywhere to talk to someone that has one, or find results, but here and a few GitHub repos are the most information I've been able to find so far.
I've been looking to help validate some of my ideas and decision, and so far I found enough to valid getting my 1100, but not enough to risk getting a 1550 OAM, unless by some luck I can find PCIe 1550.

 

[Civiloid]

Question, where are you able to find this information?

I was just asking people on discord servers that were related to hardware, and in one of them (I don't remember which one) got that reply. That was quite some time ago as well.

And from what I understand with OAM you basically need to buy a chassis that has OAM, at least for now.

 

[charlatanshost]

I was just asking people on discord servers that were related to hardware, and in one of them (I don't remember which one) got that reply. That was quite some time ago as well.

And from what I understand with OAM you basically need to buy a chassis that has OAM, at least for now.

Ah gotcha, I appreciate the information.

Yeah that's what I've found about OAM so far, but I would like to confirm it, but I would like to find a production OAM 1550 at a decent price.

 

[jo2305]

Have you guys considered Gaudi 2? They are really cheap on ebay right now, although 96gb VRAM not 128gb. But they are official pulls not ES. Still have to deal with the OAM headache

 

[charlatanshost]

Have you guys considered Gaudi 2? They are really cheap on ebay right now, although 96gb VRAM not 128gb. But they are official pulls not ES. Still have to deal with the OAM headache

I've considered them, been looking into them. I got confirmation from the guy that make the single card OAM to PCIe adapters, he said his clients confirmed it worked with theirs.
But still waiting for the prices to come down a little more before I get one.

 

[jo2305]

I've considered them, been looking into them. I got confirmation from the guy that make the single card OAM to PCIe adapters, he said his clients confirmed it worked with theirs.
But still waiting for the prices to come down a little more before I get one.

Wow, that's great news. I was really discouraged that nobody was able to get the MI250X OAM working with it. But I'll take that!

 

[bayleyw]

I've considered them, been looking into them. I got confirmation from the guy that make the single card OAM to PCIe adapters, he said his clients confirmed it worked with theirs.
But still waiting for the prices to come down a little more before I get one.

As far as I know, no up-to-date software framework supports Ponte Vecchio or Gaudi 2 (Intel has essentially abandoned both after financial pressure last year).

 

[jo2305]

As far as I know, no up-to-date software framework supports Ponte Vecchio or Gaudi 2 (Intel has essentially abandoned both after financial pressure last year).

Well there is an SDK for Habana/Gaudi, the latest one isn't open source but was released in April and still usable. Also I think Intel is integrating Gaudi into Jaguar Shores.

 

[bayleyw]

Well there is an SDK for Habana/Gaudi, the latest one isn't open source but was released in April and still usable. Also I think Intel is integrating Gaudi into Jaguar Shores.

yeah, an SDK for all those times you want to hand-implement sparse attention and MoE kernels on an architecture where the only known code is from the vendor. a truly "educational experience" to be sure.

no one knows what Jaguar Shores will be like. it will probably use some Gaudi IP, but my guess is it will have a more normal layout. Gaudi was this weird TPU-style design revolving around a few big systolic arrays.