Edit: I misread the specs and posted this a bit too hastily. I'd delete this comment but it already has children.
cosmotic
I think these are totally different spaces. Th OP is for an AI dev machine. Framework Desktop is general purpose.
jauntywundrkind
They both have ~0.25TBps memory bandwidth & huge GPUs. They very much are in the same space.
I'm very Team Red (AMD) but I dig & respect the 4x 40Gbps USB ports here. Its not popular but it is a good way to attach pretty fast storage. Also the 200Gbps connect-x7 is just awesome. Users can add what they will to Strix Halo, but it's neat seeing good connectivity on chip.
I hope AMD can get better about USB4 & Thunderbolt 4. It really feels like it should be part of their APUs. Apple shipping 6x TB5 ports on the new M3 Ultra is a beautiful example of what life should be like.
rbanffyOP
> They very much are in the same space.
I would place the Asus a peg above the Framework in both CPU and GPU. It also has a bit higher memory bandwidth. All in all, it's pretty close, but what really sets it apart from anything else that size is the network connectivity: one 10Gbe port and a pair of X7 connectors intended for clustering.
> Apple shipping 6x TB5 ports on the new M3 Ultra is a beautiful example of what life should be like.
AMD is well positioned to build specialized products if this niche proves profitable. I can easily imagine them making cut down versions of their MI300A with varying numbers of CPU and GPU tiles.
What I dislike about this is the size. They could make it a bit larger, which would, unavoidably, make it quieter as well.
sliken
Sort of. Sure you can game on the framework desktop, but I'd expect them to do similar on inference workloads and come with the same max ram and bandwidth.
rbanffyOP
Any computer is general purpose if you want it hard enough.
I hope this being more accessible than other Nvidia gear allows it to develop a healthy software ecosystem that doesn't depend so much on Nvidia.
sliken
1000 TOPS sounds good, until you notice the 273GB/sec. This will allow larger models than consumer GPUs, but be rather slow running them.
eckelhesten
>Any computer is general purpose if you want it hard enough.
True, but not any general purpose computer comes with 1000TOPS of computational power.
nashashmi
The coolest part is the thing is powered by a USB C. Granted it is not an intel, but the first powerful mini pc that does not have a separate dc power port is a breath of fresh air .
rbanffyOP
It also means there will be a power brick under the desk. I'm not sure there are monitors that can power it from their USB-C ports.
zokier
At 170 watts? Unlikely that there are many monitors with that level of USB-PD.
rbanffyOP
I wonder if you can attach it to two monitors and power it that way. I don't think anyone expects the user of such a beast to have a single screen.
anonymousiam
There are plenty of docks that will supply 100W via USB-C PD.
rbanffyOP
This draws 170. Not ridiculous, but quite a lot. And the dock can still count as a power brick.
simoncion
> The coolest part is the thing is powered by a USB C. [This is] a breath of fresh air.
Serious question, looking for a serious answer from you: Why?
The wafer-style USB-C connector is mechanically inferior to the size of barrel-style connectors that are typically used for 100+W applications. It's far less resistant to torque and shear, and due to its small size is significantly easier to accidentally dislodge. While accidental disconnection isn't a huge problem for a battery-powered machine, this thing is wall-powered.
Making things worse, unless you want to roll the dice on Amazon not sending you something that might burn your house down due to underspecced conductors and/or connectors, you're going to have to pay a significant fraction of the cost of the PSU for the cable to connect it.
There are also the ergonomic factors to consider. On the overwhelming majority of consumer hardware that's powered with a barrel-style connector, there's only one place where that connector fits. However, on much-to-most of the consumer hardware that's powered with a USB-C connector, there are many plugs that will accept the power supply cable but will refuse to use it to power the hardware. Perhaps one day in the distant future, manufacturers will universally pay the money to make all USB-C-shaped plugs capable of powering the USB-C-connector-powered-hardware they're built in to, but that day is not today.
And sure, you can reasonably retort with "Well, you just keep trying plugs until it works.", as well as "Well, just turn on a light and read the printing on the plugs.". These aren't huge hassles. But I remember that one of the big hassles that the USB-C connector saved us from was "You don't have to try to plug it in three times to get it right.". Going from "Well, it just works, first try!" to "Well, you just have to find the right plug to make it work. Keep trying!" is backwards and a little sad.
nashashmi
A use case is portability. I can transport the device between dock equipped workstations. Having to plug and unplug just one port vs 2 is a lot easier.
I am not concerned with durability of the port. My experience with one of many usb port suited for high power delivery has been uneventful. I know my devices well enough.
The only drawback is insufficient power delivery from docks. It’s only necessary to me that it works with less power.
simoncion
Calling "being able to detach and reattach one cable" (assuming that the external power supply you're planning to use can actually handle the power and data transfer job you're asking of it, so really this is a capability which just isn't going to work some-to-much of the time) a "breath of fresh air" is pretty seriously overstating things.
Especially when you're going to have a hard time powering a powerful small machine... 120->240W doesn't provide a lot of grunt.
jokethrowaway
My RPI is powered via USB-C but if it's the wrong cable or the wrong power source then it's useless; the device will boot and then restart immediately
Pretty useless standard if the specs can vary so much
simoncion
On the one hand, if you plug in a power adapter that doesn't provide enough power to most anything, the powered device will malfunction.
On the other hand, it's often the case with high-powered devices that power adapters that won't provide enough power simply won't fit in the receptacles of devices that need more power.
(On the OTHER other hand, it's very rare [0] that a power adapter will ship with a cable that isn't rated for the load that the adapter is rated for.)
[0] Well, unless you're buying drop-shipped trash from Amazon... but in that case, the whole damn assembly is probably a serious fire and/or electrocution hazard.
RamboRogers
Memory speed is way slower than mac studio, be interested to see benchmarks.
bitsandboots
Benchmarks of what? Memory speed matters for some things but not others. It matters a lot for AI training, but less for AI inference. It matters a lot for games too, but nobody would play a game on this or a mac.
buildbot
AI inference is actually typically bandwidth limited compared to training, which can re-use the weights for all tokens <sequence length> * <batch size>. Inference, specifically decoding, requires you to read all of the weights for each new token, so the flops per byte are much lower during inference!
fancyfredbot
Looks to be a smart play from NVIDIA to recapture the local LLM market, this looks very competitive with Apple M4 and AMD AI Max, but cheaper(!).
Nvidia and cheap don't go together most of the time so
I think they must have been very worried by developers and enthusiasts buying other hardware.
Consistent with their very developer focused strategy and probably a very smart idea since it's low enough spec to avoid canabalising sales of their existing products.
fancyfredbot
Too late to edit my post but wanted to apologise for causing confusion.
I suppose "cheapest" can be a very subjective term if the comparison is between things with different capabilities.
NVIDIA is a cheaper option than AMD only if you compare assume you want/need the fast networking NVIDIA are bundling with their system. According to the article the NIC would add $1500-$2000 to the price of another systems. I also failed to account for the extra memory bandwidth offered by M4 max. The apple system costs more but if you want/need that bandwidth then it's the cheapest of the three.
I guess "the system has a niche where it offers very good price/performance" is what I should have said. Not as snappy though.
bitsandboots
I'd say the SGX is the "cheapest" only if you're trying to go beyond 96GB.
the AMD system is 96GB max for the GPU. The 128GB allocates a minimum of 32GB to the CPU.
the Nvidia system is designed to be connected to a second if so desired, making it the cheapest way to get 256GB.
If you're just going for something under 96GB, haven't seen something cheaper than the AMD system for anything that can't fit on a traditional GPU. And even then, GPUs are obscene ripoff prices lately. Here's hoping these won't be scalped too.
ezschemi
The 96GB is the maximum on Windows. On Linux, you can allocate 110GB.
tracker1
AMD AI Max+ 395 systems should be around $2k with 128gb DDR5X, the Mac Studio maxed out is just north of $12k. So it's in the middle depending. This will also vary greatly in terms of usability for other things.
That said, a nice OEM option if you need something for AI workloads where the GPU market is completely soaked with scalpers. Been considering a drive to California just so I can get a GPU directly at MicroCenter instead of paying scalper overheads.
sliken
Yes, you can spend $12k on a mac studio. However the Mac studio with the m4 max has 128GB ram, double the memory bandwidth, and costs $3,699. LESS than the DGX spark. Granted it doesn't have 200gbe.
If it's the memory bandwidth you are after the Mac Mini with the m4 pro has similar, but max 64GB ram.
fancyfredbot
Article is the ASUS system which is $3000, so cheaper than the M4 max.
sliken
Indeed, 23% more money for double the memory bandwidth.
rbanffyOP
> This will also vary greatly in terms of usability for other things.
This is key. Nvidia has a terrible reputation with long term support (as market leaders, they can easily afford that). Apple just now (last November) dropped OS updates for their 2014 boxes. While a Mac Studio 2025 will not be a ridiculous amount of compute power in 10 years, I fully expect Nvidia to completely abandon support and software updates for this in five years tops.
Hopefully, considering the interest it generated, I'd hope the Linux crowd will be able to carry it further, maybe with decent open-source drivers, way past the expiration date Nvidia sets.
tracker1
That'll be the hard part for sure... NVidia is in a position to want to push people to abandon older tech for new shiny. I would hope to see these machines last a decade all the same. Also interested to see how the level of compute compares to other pro and consumer options.
bitsandboots
> Nvidia has a terrible reputation with long term support
In what space do they have this reputation? In drivers, I see they're supporting hardware that's 10 years old right now.
scottapotamas
Their single board computers intended for robotics/edge have had a history of being poorly supported and stuck on old kernel versions.
Cheaper? The DGX spark is $3,000 with 128GB ram. A framework desktop with the AMD strix halo 395 with 128gb ram is $2,000 and has better memory bandwidth. No price I can see for the identically spec'd (and nearly identical looking) Ascent gx10.
[edit] Oops, the Spark is $4,000, only the Ascent is at $3k now. Strix Halo systems vary from slightly slower (6%) to the same (on systems with LPDDR5x-8533, like the HP laptop).
walterbell
Includes RDMA 200GbE NIC.
> NVIDIA ConnectX-7 NIC these days often sells for $1500-2200 in single unit quantities, depending on the features and supply of the parts. At $2999 for a system with this buit-in that is awesome.
kcb
Yea unlike other options this is actually scalable. The question isn't if 1 can outperform the Mac Studio but if 3-4 linked together can.
Dylan16807
What kind of scaling do you have in mind there?
My naive analysis is: A high end Mac should be able to run each layer of an AI task about twice as fast because of the memory bandwidth. And the data going between layers is tiny enough to run over thunderbolt or even normal ethernet.
Is there an AI use case that prefers 250GB/s memory bandwidth plus 25GB/s interconnect over 500GB/s memory and 2GB/s interconnect? Are there other major use cases that prefer it?
fancyfredbot
Usually the reason you'd want the network bandwidth would be for distributed training.
For inference you can probably get by with 2GB/s assuming you can split the layers up nicely.
The interconnect can be a bottleneck for inference but only for networks with loads of activations and large batch sizes, or if you are doing tensor level parallelism.
sliken
Sort of. They come in different speeds, and those prices are for the 400gbe version. The 200gbe, like in the GX10 and spark are more like $1250. Not to mention you have to need that the 200gbe (to cluster 2 of them) and I'd expect the vast majority to buy a single unit, not a pair.
walterbell
With a crossover cable, a single unit could be used for local testing of software that depends on both RDMA and CUDA.
wtallis
Double check your memory bandwidth numbers. AMD's Strix Halo is 256 bits at 8000MT/s for about 238GB/s while NVIDIA's GB10 is 276GB/s (likely 256 bits at the more standard speed of 8533MT/s).
sliken
Depends which one, the HP ZBook Ultra uses LPDDR5x-8533, a dead match for what nvidia claims (273GB/sec). Although the DGX spark now costs $4,000 instead of the "starting at $2,999" mentioned a few months ago.
So the bandwidth is dead even between AMD and Spark.
wtallis
The HP ZBook Ultra is using DRAM parts rated for 8533MT/s but only operating at 8000MT/s, because 8000MT/s is the most the processor is rated for, but the memory manufacturers don't make parts with that non-stamdard speed grade.
sliken
Ah, thanks, I didn't know that.
rubatuga
Agreed not cheaper by a long shot.
Tepix
Do the math. A dense 100GB model will be awfully slow at 273GB/s mem bandwidth.-
So far, project digits looks disappointing.
fancyfredbot
A 100GB model running on one of these would be a disaster, you'd probably run out of space for the KV cache apart from anything else. But if you had two of them performance should be acceptable.
I think they would be canabalising their other product lines if they had more memory bandwidth.
- [0] https://frame.work/desktop
Edit: I misread the specs and posted this a bit too hastily. I'd delete this comment but it already has children.
I'm very Team Red (AMD) but I dig & respect the 4x 40Gbps USB ports here. Its not popular but it is a good way to attach pretty fast storage. Also the 200Gbps connect-x7 is just awesome. Users can add what they will to Strix Halo, but it's neat seeing good connectivity on chip.
I hope AMD can get better about USB4 & Thunderbolt 4. It really feels like it should be part of their APUs. Apple shipping 6x TB5 ports on the new M3 Ultra is a beautiful example of what life should be like.
I would place the Asus a peg above the Framework in both CPU and GPU. It also has a bit higher memory bandwidth. All in all, it's pretty close, but what really sets it apart from anything else that size is the network connectivity: one 10Gbe port and a pair of X7 connectors intended for clustering.
> Apple shipping 6x TB5 ports on the new M3 Ultra is a beautiful example of what life should be like.
AMD is well positioned to build specialized products if this niche proves profitable. I can easily imagine them making cut down versions of their MI300A with varying numbers of CPU and GPU tiles.
What I dislike about this is the size. They could make it a bit larger, which would, unavoidably, make it quieter as well.
I hope this being more accessible than other Nvidia gear allows it to develop a healthy software ecosystem that doesn't depend so much on Nvidia.
True, but not any general purpose computer comes with 1000TOPS of computational power.
Serious question, looking for a serious answer from you: Why?
The wafer-style USB-C connector is mechanically inferior to the size of barrel-style connectors that are typically used for 100+W applications. It's far less resistant to torque and shear, and due to its small size is significantly easier to accidentally dislodge. While accidental disconnection isn't a huge problem for a battery-powered machine, this thing is wall-powered.
Making things worse, unless you want to roll the dice on Amazon not sending you something that might burn your house down due to underspecced conductors and/or connectors, you're going to have to pay a significant fraction of the cost of the PSU for the cable to connect it.
There are also the ergonomic factors to consider. On the overwhelming majority of consumer hardware that's powered with a barrel-style connector, there's only one place where that connector fits. However, on much-to-most of the consumer hardware that's powered with a USB-C connector, there are many plugs that will accept the power supply cable but will refuse to use it to power the hardware. Perhaps one day in the distant future, manufacturers will universally pay the money to make all USB-C-shaped plugs capable of powering the USB-C-connector-powered-hardware they're built in to, but that day is not today.
And sure, you can reasonably retort with "Well, you just keep trying plugs until it works.", as well as "Well, just turn on a light and read the printing on the plugs.". These aren't huge hassles. But I remember that one of the big hassles that the USB-C connector saved us from was "You don't have to try to plug it in three times to get it right.". Going from "Well, it just works, first try!" to "Well, you just have to find the right plug to make it work. Keep trying!" is backwards and a little sad.
I am not concerned with durability of the port. My experience with one of many usb port suited for high power delivery has been uneventful. I know my devices well enough.
The only drawback is insufficient power delivery from docks. It’s only necessary to me that it works with less power.
Especially when you're going to have a hard time powering a powerful small machine... 120->240W doesn't provide a lot of grunt.
Pretty useless standard if the specs can vary so much
On the other hand, it's often the case with high-powered devices that power adapters that won't provide enough power simply won't fit in the receptacles of devices that need more power.
(On the OTHER other hand, it's very rare [0] that a power adapter will ship with a cable that isn't rated for the load that the adapter is rated for.)
[0] Well, unless you're buying drop-shipped trash from Amazon... but in that case, the whole damn assembly is probably a serious fire and/or electrocution hazard.
Nvidia and cheap don't go together most of the time so I think they must have been very worried by developers and enthusiasts buying other hardware.
Consistent with their very developer focused strategy and probably a very smart idea since it's low enough spec to avoid canabalising sales of their existing products.
I suppose "cheapest" can be a very subjective term if the comparison is between things with different capabilities.
NVIDIA is a cheaper option than AMD only if you compare assume you want/need the fast networking NVIDIA are bundling with their system. According to the article the NIC would add $1500-$2000 to the price of another systems. I also failed to account for the extra memory bandwidth offered by M4 max. The apple system costs more but if you want/need that bandwidth then it's the cheapest of the three.
I guess "the system has a niche where it offers very good price/performance" is what I should have said. Not as snappy though.
the AMD system is 96GB max for the GPU. The 128GB allocates a minimum of 32GB to the CPU.
the Nvidia system is designed to be connected to a second if so desired, making it the cheapest way to get 256GB.
If you're just going for something under 96GB, haven't seen something cheaper than the AMD system for anything that can't fit on a traditional GPU. And even then, GPUs are obscene ripoff prices lately. Here's hoping these won't be scalped too.
That said, a nice OEM option if you need something for AI workloads where the GPU market is completely soaked with scalpers. Been considering a drive to California just so I can get a GPU directly at MicroCenter instead of paying scalper overheads.
If it's the memory bandwidth you are after the Mac Mini with the m4 pro has similar, but max 64GB ram.
This is key. Nvidia has a terrible reputation with long term support (as market leaders, they can easily afford that). Apple just now (last November) dropped OS updates for their 2014 boxes. While a Mac Studio 2025 will not be a ridiculous amount of compute power in 10 years, I fully expect Nvidia to completely abandon support and software updates for this in five years tops.
Hopefully, considering the interest it generated, I'd hope the Linux crowd will be able to carry it further, maybe with decent open-source drivers, way past the expiration date Nvidia sets.
In what space do they have this reputation? In drivers, I see they're supporting hardware that's 10 years old right now.
[edit] Oops, the Spark is $4,000, only the Ascent is at $3k now. Strix Halo systems vary from slightly slower (6%) to the same (on systems with LPDDR5x-8533, like the HP laptop).
> NVIDIA ConnectX-7 NIC these days often sells for $1500-2200 in single unit quantities, depending on the features and supply of the parts. At $2999 for a system with this buit-in that is awesome.
My naive analysis is: A high end Mac should be able to run each layer of an AI task about twice as fast because of the memory bandwidth. And the data going between layers is tiny enough to run over thunderbolt or even normal ethernet.
Is there an AI use case that prefers 250GB/s memory bandwidth plus 25GB/s interconnect over 500GB/s memory and 2GB/s interconnect? Are there other major use cases that prefer it?
For inference you can probably get by with 2GB/s assuming you can split the layers up nicely.
The interconnect can be a bottleneck for inference but only for networks with loads of activations and large batch sizes, or if you are doing tensor level parallelism.
So the bandwidth is dead even between AMD and Spark.
So far, project digits looks disappointing.
I think they would be canabalising their other product lines if they had more memory bandwidth.