Yea unlike other options this is actually scalable. The question isn't if 1 can outperform the Mac Studio but if 3-4 linked together can.
What kind of scaling do you have in mind there?
My naive analysis is: A high end Mac should be able to run each layer of an AI task about twice as fast because of the memory bandwidth. And the data going between layers is tiny enough to run over thunderbolt or even normal ethernet.
Is there an AI use case that prefers 250GB/s memory bandwidth plus 25GB/s interconnect over 500GB/s memory and 2GB/s interconnect? Are there other major use cases that prefer it?
Usually the reason you'd want the network bandwidth would be for distributed training.
For inference you can probably get by with 2GB/s assuming you can split the layers up nicely.
The interconnect can be a bottleneck for inference but only for networks with loads of activations and large batch sizes, or if you are doing tensor level parallelism.
Sort of. They come in different speeds, and those prices are for the 400gbe version. The 200gbe, like in the GX10 and spark are more like $1250. Not to mention you have to need that the 200gbe (to cluster 2 of them) and I'd expect the vast majority to buy a single unit, not a pair.
> NVIDIA ConnectX-7 NIC these days often sells for $1500-2200 in single unit quantities, depending on the features and supply of the parts. At $2999 for a system with this buit-in that is awesome.