- angoragoatsI have a 25+ year career history with lots of contacts, and I deleted my LI account years ago. I do not regret it one bit; in fact I wish I had done it sooner.
- LinkedIn is not a social network, it’s a vehicle for spam, grifting, self-promotion and other useless garbage. I deleted my account years ago and I wish I had done it sooner.
- I have been told that I’m obstinate before :)
To be fair though, I don’t really want a Big Tech job. Several of the FAANGs, especially Facebook, are morally objectionable to me and I would switch careers before working for them. Most others have shitty working conditions with in-office policies, open office layouts, etc, that are detrimental to me getting work done.
So it’s not just about the financial RoI for me.
And I think I’m at least consistent: I’ve never been one to jump through hoops for raises or promotions either.
- This is a good way to frame it. I have no issue with people who choose to do this, but I choose not to.
- No, I don't, assuming you're looking at total comp including stock.
I also don't have crazy hours, stack ranking, mandatory RTO, 6 different levels of OKRs, and all the other bullshit that comes along with a FAANG job.
- FWIW you do not have to do leetcode to get a job. I’m a staff level SWE and I’ve had no problems finding non-leetcode companies, both through my own network and through browsing job listings.
I have literally never done a leetcode “whiteboard code this random problem that has nothing to do with our actual work” interview, in 25+ years in the industry.
- Ok. I’ve never seen a configuration like this, while using breakout cables to go from higher bandwidth -> multiple lower bandwidth clients is common, so I still disagree with your assertion that it seems “more likely” that this would be supported.
- You’re talking about link aggregation (LACP) here, which requires specific settings on both the switch and client machine to enable, as well as multiple ports on the client machine (in your example, multiple 50Gbps ports). So while it’s likely possible to combine 50Gbps ports like you describe, that’s not what I was referring to.
- MacOS does not support RoCE.
- Here’s an example of the cables I was referring to that can split a single 400Gbit QSFP56-DD port to two 200Gbit ports:
https://www.fs.com/products/101806.html
But all of this is pretty much irrelevant to my original point.
- The OP makes reference to this with a link to a GitHub repo that has some benchmarks. TCP over Thunderbolt compared to RDMA over Thunderbolt has roughly 7-10x higher latency, ~300us vs 30-50us. I would expect TCP over 200GbE to have similar latency to TCP over Thunderbolt.
Put another way, see the graphs in the OP where he points out that the old way of clustering performs worse the more machines you add? I’d expect that to happen with 200GbE also.
And with a switch, it would likely be even worse, since the hop to the switch adds additional latency that isn’t a factor in the TB5 setup.
- As with most 40+GbE ports, the 400Gbit ports can be split into 2x200Gbit ports with the use of special cables. So you can connect a total of 6 machines at 200Gbit.
- Cool! So for marginally less in cost and power usage than the numbers I quoted, you can get 2 more machines than with the RDMA setup. And you’ve still not solved the thing that I called out as the most important drawback.
- > Also, as the OP noted, this setup can support up to 4 Mac devices because each Mac must be connected to every other Mac!! All the more reason for Apple to invest in something like QSFP.
This isn’t any different with QSFP unless you’re suggesting that one adds a 200GbE switch to the mix, which:
* Adds thousands of dollars of cost,
* Adds 150W or more of power usage and the accompanying loud fan noise that comes with that,
* And perhaps most importantly adds measurable latency to a networking stack that is already higher latency than the RDMA approach used by the TB5 setup in the OP.
- If you travel to China, sure, what I’m talking about probably won’t work for you.
In pretty much any other situation, using dedicated GPUs is 1) definitely faster, like 2x the speed or more depending on your use case, and 2) the same cost or possibly cheaper. That’s all I’m saying.
- Sure, the GPUs sit in my basement and I can connect to them from anywhere in the world.
My point was not that “it isn’t really that slow,” my point is that Macs are slower than dedicated GPUs, while being just as expensive (or more expensive, given the specific scenario) to purchase and operate.
And I did my analysis using the Mac Studio, which is faster than the equivalent MBP at load (and is also not portable). So if you’re using a MacBook, my guess is that your performance/watt numbers are worse than what I was looking at.
- You illustrated my point exactly: yes, a single 32GB 5090 has half the memory of your Mac. But two of them (or three 3090/4090s) have the same total memory as your Mac, are in the same ballpark in price, and would be several times faster at running the same model as your Mac.
And before you bring up the “efficiency” of the Mac: I’ve done the math, and between the Mac being much slower (thus needing more time to run) and the fact that you can throttle the discrete GPUs to use 200-250W each and only lose a few percent in LLM performance, it’s the same price or cheaper to operate the discrete GPUs for the same workload.
- Hard disagree on it working well for local AI - all the memory bandwidth in the world doesn’t matter when the GPU it’s connected to is middling in performance compared to dedicated options. Give me one (or several) 3090/4090/5090 any day of the week over a Mac.
- What the heck does “produce alignment” mean? I don’t produce alignment, I produce software which solves problems for people.
- I wish more people working for these companies with short-sighted RTO mandates would explore suing their employer for promissory estoppel. There were real guarantees along the lines of “in-office work is never coming back” made verbally by CEOs when COVID began. If employees made financial/legal decisions based on these false promises, the company should be held liable.