For some additional context, I believe most cloud providers who started on FPGAs for network devices have switched to “smart” ASICs, where some limited portion of the device is programmable.
The issue with FPGAs, is that if you have enough scale, an ASIC starts to make more sense.
Still, I believe they provide a glimpse into the hardware of the future!
Same here. I believe that the lack of programmable hardware is akin to the lack of high multicore CPUs. As in, either would be too disruptive to the status quo.
GPU/TPU etc are just domain-specific hardware, not much help for exploring new paradigms.
To really solve this, we'll likely need someone who's won the internet lottery and is willing to invest serious capital ($1 billion plus) to solve real problems and get real work done. Until then, it's going to be tech bros playing with VR and LLMs.
I'll see you all in 10 years when the situation still hasn't changed.
For me, the promise of computation is not in the current CPU (or now GPU) world, but one in which the hardware can dynamically run the most optimized logic for a given operation.
Sadly, this has yet to materialize (with some exceptions[0][1][2]).
Hopefully in an LLM-first world, we can start to utilize FPGAs more effectively at (small) scale :)
[0]https://www.microsoft.com/en-us/research/wp-content/uploads/...
[1]https://novasparks.com/
[2]https://www.alibabacloud.com/blog/deep-dive-into-alibaba-clo...