most of them are much more general purpose. they might be specializing somewhat on the architecture, but not on the weights
Realistically specializing on the data flow is all you can do. Assuming a modern CPU contains on the order of 10 billion transistors that only amounts to 1.2 GiB storage before you account for any actual logic (ie 1 bit per transistor). DRAM hardware is quite different from that of processing elements and it takes quite a lot of DRAM chips to hold the weights of a single model.
More seriously, isn't that pretty much what all those AI hardware startups have already been doing for a while now?