> There is no reason why we should assume a silicon machine must have the same capabilities as a carbon machine
Then make your computer out of carbon.
While the broader principle, that we don't know what we're doing and AI as it currently exists is a bit cargo-culty, this is a critique of the SOTA and is insufficient to be generalised: we can reasonably say "we probably have not", we can't say "we definitely cannot ever".
Who knows, perhaps our brains do somehow manage to do whacky quantum stuff despite seeming to be far too warm and messy for that. But even that is just an implementation detail.
Unless you can show - even a single example would do - that we can compute a function that is outside the Turing computable set, then there is a very strong reason that we should assume a silicon machine has the same capabilities as a carbon machine to compute.
The problem is that your challenge is begging the question.
Computability or algorithms are the problem.
It is all the 'no effective algorithm exists for X' that is the problem.
Spike train retiming and issues with riddled basins in existing computers and math is an example if you drop compute a function
Iron and copper are both metals but only one can be hardened into steel
There is no reason why we should assume a silicon machine must have the same capabilities as a carbon machine