Hacker Neue

Preferences
dnhkng parent
This is the story of how I bought enterprise-grade AI hardware designed for liquid-cooled server racks that was converted to air cooling, and then back again, survived multiple near-disasters (including GPUs reporting temperatures of 16 million degrees), and ended up with a desktop that can run 235B parameter models at home. It’s a tale of questionable decisions, creative problem-solving, and what happens when you try to turn datacenter equipment into a daily driver.
amirhirsch
# Tell the driver to completely ignore the NVLINK and it should allow the GPUs to initialise independently over PCIe !!!! This took a week of work to find, thanks Reddit!

I needed this info, thanks for putting it up. Can this really be an issue for every data center?

Tinyyy
Doesn’t this prevent the GPUs from talking to each other over the high speed link?
dnhkng OP
I'll find out soon, but without this hack, the GPUs are non-functional.
ipsum2
I saw the same post on Reddit and was so tempted to purchase it, but I live in the US. Cool to see it wasn't a scam!
GPTshop
We can get around tariffs, if that is your concern.
ipsum2
Honestly I wasn't going to drop ~10k USD on an unknown seller that was from another country.
GPTshop
There is always some risk in business and life itself...
pointbob
Loved it. You are mgyver. You should post more stuff on Twitter. Thanks for the story.
dnhkng OP
lol, I tried posting stuff on Twitter, but never got any traction. This might be too nerdy for that crowd?
DANmode
Mastodon and Bluesky would welcome you.

Hackaday would probably welcome you.

dauertewigkeit
It's a very interesting read, but a lot is not clear.

How does the seller get these desktops directly from NVIDIA?

And if the seller's business is custom made desktop boxes, why didn't he just fit the two H100s into a better desktop box?

Ntrails
> why didn't he just fit the two H100s into a better desktop box?

I expect because they were no longer in the sort of condition to sell as new machines? They were clearly well used and selling "as seen" is the lowest reputational risk associated with offload

wtallis
There also weren't H100s available to scavenge. GH200 puts the Grace CPU and H100 GPU on a big module with a custom form factor and connectors, so the only viable route for using those GPUs was to keep all the electronics together and build a suitable case and cooling system around them. There wasn't any way to adapt any of this for use in an ordinary EATX case or with a different CPU, because the GPUs weren't PCIe add-in cards.
renewiltord
At that pricing I honestly thought they fell off a truck. Even well used H100 go for more than that entire system. In the US an RTX A6000 Ada is already close in price.
dnhkng OP
These are on a custom board from Nvidia, so its not possible to separate them. I think the seller usually gets H100's and them into a custom case, with a PCIE adapter to the server GPUs.

This thing too unwieldy to make into a desktop (you can see how much effort it took), and was in pretty bad condition. I think he just wanted to get rid of it without having to deal with returns. I took a bet on it, and was lucky it paid out.

GPTshop
We build these desktops from Nvidia servers we buy from reputable manufacturers like Pegatron, Gigabyte, Asrock Rack, and many more.

H100 PCI and GH200 are two very different things. The advantages of Grace Hopper are much higher connections speeds, bandwidth and lower power consumption.

ProAm
Which is how you learn to become an expert. I love it
baud147258
When you said you paid cash, you paid all ~7.5k€ in paper money? How do you get that much cash out of your bank?
devilbunny
Presumably by going there, showing your ID, and withdrawing it? They might make you wait a day to have that much on hand, but not more than that.
leipert
We are talking Germany here. People buy cars in cash. You don’t even have to necessarily wait a day.
devilbunny
True, which is why I said “might”. Even in the US. I only have to call ahead if I want smaller bills - $20 and $100 they usually have plenty of unless it’s a tiny branch.
baud147258
ok, I guess. I just never handled that many. I would also have though that the bank might ask questions
devilbunny
Cash deposits or withdrawals over $10k in the US will be reported to the Treasury but if you don’t do them often it won’t raise a big flag, and the bank doesn’t care what you do with it. Treasury only cares if they think you are trying to evade taxes.

In Germany, where large items are often purchased with cash, it would be unremarkable if you did it several times a year.

dnhkng OP
Sure, its a free country! (I'm Australian, living in Germany).
jerome-jh
Securing soldered components with epoxy? You have to be very confident at your soldering :) You had no hot glue?
dnhkng OP
I've had a bit of practice, but I don't have the right gear for this level of soldering. It took maybe an hour to solder in 2 components, after many failed attempts. Persistence beats intelligence?
Fire-Dragon-DoL
Did it behave like a star at 16 million degrees? Lol

This item has no comments currently.

Keyboard Shortcuts

Story Lists

j
Next story
k
Previous story
Shift+j
Last story
Shift+k
First story
o Enter
Go to story URL
c
Go to comments
u
Go to author

Navigation

Shift+t
Go to top stories
Shift+n
Go to new stories
Shift+b
Go to best stories
Shift+a
Go to Ask HN
Shift+s
Go to Show HN

Miscellaneous

?
Show this modal