I think to extend on the question, though, the fundamental answer is "There is nothing stopping the LLM from containing the embedding of all basic math", with the proviso that tokenization makes it vanishingly unlikely (perhaps in the current generation, or within reasonable resource limits).

I see it analogous to asking a human why they don't just "learn all the answers to simple arithmetic involving integers below 10,000" - you possibly could, it would just be a huge waste of time when you can instead learn the algorithm directly. Of course, LLMs are inherently a layer on top of an existing system which solves those problems quite well already, so it'd be somewhat silly there too.