Hacker Neue

Preferences
marcosdumay parent
Yeah, make the network deeper.

When all you have is a hammer... It makes a lot of sense that a transformation layer that makes the tokens more semantically relevant will help optimize the entire network after it and increase the effective size of your context window. And one of the main immediate obstacle stopping those models from being intelligent is context window size.

On the other hand, the current models already cost something on the line of the median country GDP to train, and they are nowhere close to that in value. The saying that "if brute force didn't solve your problem, you didn't apply enough force" is intended to be listened as a joke.

jagraff
I think the median country GDP is something like $100 Billion

https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(PPP)

Models are expensive, but they're not that expensive.

telotortium
LLM model training costs arise primarily from commodity costs (GPUs and other compute as well as electricity), not locally-provided services, so PPP is not the right statistic to use here. You should use nominal GDP for this instead. According to Wikipedia[0], the median country's nominal GDP (Cyprus) is more like $39B. Still much larger than training costs, but much lower than your PPP GDP number.

[0] https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nomi...

kordlessagain
The median country GDP is approximately $48.8 billion, which corresponds to Uganda at position 90 with $48.769 billion.

The largest economy (US) has a GDP of $27.7 trillion.

The smallest economy (Tuvalu) has a GDP of $62.3 million.

The 48 billion number represents the middle point where half of all countries have larger GDPs and half have smaller GDPs.

hoseja
Well then you have to agree that $48.8 billion IS "something like $100 billion".
marcosdumay OP
$100 billion is the best estimate around of how much OpenAI took in investment to build ChatGPT.
Eisenstein
A top of the line consumer desktop, the Mac Pro, costs $7000. The commonly acknowledged first non-mechanical computer, the ENIAC, cost $400,000, which adjusted for inflation is $6,594,153 (see note). Will AI models follow the same pricing trajectory? Probably not but they no longer cost even close to $100 billion.

Note: 1946 CPI = 19.5, 2025 CPI = 321.465 which makes for an increase of 16.49.

ayewo
It seems all you did was use this formula:

CPI{2025} / CPI{1946} * Price{1946} = Price{2025}

to obtain the price adjusted for inflation?

That is the only way I was able to arrive at the same number you got: $6,594,153.846. TIL.

programjames
Maybe for an Australian billion? But in American English it would be $100 million.
ash_091
In both Australian and American English a billion is 1,000,000,000 (one thousand million).
amelius
Maybe it checks out if you don't use 1 year as your timeframe for GDP but the number of days required for training.
Nicook
does anyone even have good estimates for model training?
whiplash451
I get your point but do we have evidence behind “ something on the line of the median country GDP to train”?

Is this really true?

robrenaud
It's not even close.
hiddencost
I mean brute force is working great. Acceleration is large, cost per unit intelligence are dropping much faster than Moore's law. We've been doing this kind of scaling since the 80s across various measures and are quite good at it.
helloplanets
> the current models already cost something on the line of the median country GDP to train

This is just blatantly false.

> According to AI Index estimates, the training costs of state-of-the-art AI models have reached unprecedented levels. For example, OpenAI’s GPT-4 used an estimated $78 million worth of compute to train, while Google’s Gemini Ultra cost $191 million for compute.

https://hai.stanford.edu/ai-index/2024-ai-index-report

No need to even open up the actual report to find that. Just scroll down the page to read the 'key takeaways'.

This item has no comments currently.